Note: This is test shard 1 of 8.
[==========] Running 8 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from AdminCliTest
[ RUN ] AdminCliTest.RebalancerReportOnly
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 02:11:18.703323 22960 test_util.cc:268] Using random seed: 1010284718
I20240503 02:11:18.714718 22960 ts_itest-base.cc:112] Starting cluster with:
I20240503 02:11:18.714802 22960 ts_itest-base.cc:113] --------------
I20240503 02:11:18.714848 22960 ts_itest-base.cc:114] 5 tablet servers
I20240503 02:11:18.714910 22960 ts_itest-base.cc:115] 3 replicas per TS
I20240503 02:11:18.714973 22960 ts_itest-base.cc:116] --------------
2024-05-03T02:11:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:11:18Z Disabled control of system clock
I20240503 02:11:18.739284 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.108.62:37253
--webserver_interface=127.22.108.62
--webserver_port=0
--builtin_ntp_servers=127.22.108.20:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.108.62:37253 with env {}
W20240503 02:11:19.058383 22972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:19.058698 22972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:19.058878 22972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:19.066207 22972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:11:19.066341 22972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:19.066421 22972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:11:19.066500 22972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:11:19.075376 22972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.108.62:37253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.108.62:37253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.108.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:19.076718 22972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:19.078513 22972 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:19.089646 22978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:19.089915 22977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:19.090535 22980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:19.091462 22972 server_base.cc:1034] running on GCE node
I20240503 02:11:19.092149 22972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:19.093549 22972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:19.094755 22972 hybrid_clock.cc:648] HybridClock initialized: now 1714702279094696 us; error 38 us; skew 500 ppm
I20240503 02:11:19.095029 22972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:19.097251 22972 webserver.cc:460] Webserver started at http://127.22.108.62:40513/ using document root <none> and password file <none>
I20240503 02:11:19.097915 22972 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:19.098018 22972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:19.098277 22972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:19.100814 22972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "b10a2415dc36491584e3aff3c9d8df2a"
format_stamp: "Formatted at 2024-05-03 02:11:19 on dist-test-slave-sc15"
I20240503 02:11:19.101563 22972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "b10a2415dc36491584e3aff3c9d8df2a"
format_stamp: "Formatted at 2024-05-03 02:11:19 on dist-test-slave-sc15"
I20240503 02:11:19.106351 22972 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20240503 02:11:19.109406 22986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:19.110693 22972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240503 02:11:19.110823 22972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/wal
uuid: "b10a2415dc36491584e3aff3c9d8df2a"
format_stamp: "Formatted at 2024-05-03 02:11:19 on dist-test-slave-sc15"
I20240503 02:11:19.110987 22972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:19.130928 22972 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:19.131743 22972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:19.131994 22972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:19.149521 22972 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.62:37253
I20240503 02:11:19.149540 23037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.62:37253 every 8 connection(s)
I20240503 02:11:19.150910 22972 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 02:11:19.154565 23038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:19.154875 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 22972
I20240503 02:11:19.155104 22960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 02:11:19.163966 23038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b10a2415dc36491584e3aff3c9d8df2a: Bootstrap starting.
I20240503 02:11:19.166932 23038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b10a2415dc36491584e3aff3c9d8df2a: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:19.167894 23038 log.cc:824] T 00000000000000000000000000000000 P b10a2415dc36491584e3aff3c9d8df2a: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:19.170419 23038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b10a2415dc36491584e3aff3c9d8df2a: No bootstrap required, opened a new log
I20240503 02:11:19.178874 23038 raft_consensus.cc:348] T 00000000000000000000000000000000 P b10a2415dc36491584e3aff3c9d8df2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b10a2415dc36491584e3aff3c9d8df2a" member_type: VOTER last_known_addr { host: "127.22.108.62" port: 37253 } }
I20240503 02:11:19.179200 23038 raft_consensus.cc:374] T 00000000000000000000000000000000 P b10a2415dc36491584e3aff3c9d8df2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:19.179296 23038 raft_consensus.cc:729] T 00000000000000000000000000000000 P b10a2415dc36491584e3aff3c9d8df2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b10a2415dc36491584e3aff3c9d8df2a, State: Initialized, Role: FOLLOWER
I20240503 02:11:19.179929 23038 consensus_queue.cc:260] T 00000000000000000000000000000000 P b10a2415dc36491584e3aff3c9d8df2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b10a2415dc36491584e3aff3c9d8df2a" member_type: VOTER last_known_addr { host: "127.22.108.62" port: 37253 } }
I20240503 02:11:19.180205 23038 raft_consensus.cc:388] T 00000000000000000000000000000000 P b10a2415dc36491584e3aff3c9d8df2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:19.180356 23038 raft_consensus.cc:482] T 00000000000000000000000000000000 P b10a2415dc36491584e3aff3c9d8df2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:19.180531 23038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b10a2415dc36491584e3aff3c9d8df2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:19.183398 23038 raft_consensus.cc:504] T 00000000000000000000000000000000 P b10a2415dc36491584e3aff3c9d8df2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b10a2415dc36491584e3aff3c9d8df2a" member_type: VOTER last_known_addr { host: "127.22.108.62" port: 37253 } }
I20240503 02:11:19.184016 23038 leader_election.cc:302] T 00000000000000000000000000000000 P b10a2415dc36491584e3aff3c9d8df2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b10a2415dc36491584e3aff3c9d8df2a; no voters:
I20240503 02:11:19.184727 23038 leader_election.cc:288] T 00000000000000000000000000000000 P b10a2415dc36491584e3aff3c9d8df2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:19.184856 23044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b10a2415dc36491584e3aff3c9d8df2a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:19.186070 23044 raft_consensus.cc:686] T 00000000000000000000000000000000 P b10a2415dc36491584e3aff3c9d8df2a [term 1 LEADER]: Becoming Leader. State: Replica: b10a2415dc36491584e3aff3c9d8df2a, State: Running, Role: LEADER
I20240503 02:11:19.186699 23044 consensus_queue.cc:237] T 00000000000000000000000000000000 P b10a2415dc36491584e3aff3c9d8df2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b10a2415dc36491584e3aff3c9d8df2a" member_type: VOTER last_known_addr { host: "127.22.108.62" port: 37253 } }
I20240503 02:11:19.187000 23038 sys_catalog.cc:564] T 00000000000000000000000000000000 P b10a2415dc36491584e3aff3c9d8df2a [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:11:19.189707 23045 sys_catalog.cc:455] T 00000000000000000000000000000000 P b10a2415dc36491584e3aff3c9d8df2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b10a2415dc36491584e3aff3c9d8df2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b10a2415dc36491584e3aff3c9d8df2a" member_type: VOTER last_known_addr { host: "127.22.108.62" port: 37253 } } }
I20240503 02:11:19.190114 23045 sys_catalog.cc:458] T 00000000000000000000000000000000 P b10a2415dc36491584e3aff3c9d8df2a [sys.catalog]: This master's current role is: LEADER
I20240503 02:11:19.190363 23046 sys_catalog.cc:455] T 00000000000000000000000000000000 P b10a2415dc36491584e3aff3c9d8df2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b10a2415dc36491584e3aff3c9d8df2a. Latest consensus state: current_term: 1 leader_uuid: "b10a2415dc36491584e3aff3c9d8df2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b10a2415dc36491584e3aff3c9d8df2a" member_type: VOTER last_known_addr { host: "127.22.108.62" port: 37253 } } }
I20240503 02:11:19.190639 23046 sys_catalog.cc:458] T 00000000000000000000000000000000 P b10a2415dc36491584e3aff3c9d8df2a [sys.catalog]: This master's current role is: LEADER
I20240503 02:11:19.192348 23050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:11:19.197376 23050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:11:19.205633 23050 catalog_manager.cc:1347] Generated new cluster ID: 305a01abeda941889d7e345621939b21
I20240503 02:11:19.205745 23050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:11:19.238648 23050 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:11:19.240352 23050 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:11:19.258244 23050 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b10a2415dc36491584e3aff3c9d8df2a: Generated new TSK 0
I20240503 02:11:19.259143 23050 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:11:19.276165 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.1:0
--local_ip_for_outbound_sockets=127.22.108.1
--webserver_interface=127.22.108.1
--webserver_port=0
--tserver_master_addrs=127.22.108.62:37253
--builtin_ntp_servers=127.22.108.20:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240503 02:11:19.599963 23063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:19.600265 23063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:19.600366 23063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:19.607664 23063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:19.607838 23063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.1
I20240503 02:11:19.617022 23063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.108.1
--webserver_port=0
--tserver_master_addrs=127.22.108.62:37253
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.1
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:19.618386 23063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:19.620079 23063 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:19.632292 23071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:19.632897 23068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:19.633178 23069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:19.633951 23063 server_base.cc:1034] running on GCE node
I20240503 02:11:19.634546 23063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:19.635442 23063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:19.636639 23063 hybrid_clock.cc:648] HybridClock initialized: now 1714702279636585 us; error 41 us; skew 500 ppm
I20240503 02:11:19.636933 23063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:19.639130 23063 webserver.cc:460] Webserver started at http://127.22.108.1:46681/ using document root <none> and password file <none>
I20240503 02:11:19.639830 23063 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:19.639978 23063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:19.640285 23063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:19.642925 23063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b2e5237958d44a7486250a897a242ce6"
format_stamp: "Formatted at 2024-05-03 02:11:19 on dist-test-slave-sc15"
I20240503 02:11:19.643678 23063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b2e5237958d44a7486250a897a242ce6"
format_stamp: "Formatted at 2024-05-03 02:11:19 on dist-test-slave-sc15"
I20240503 02:11:19.648653 23063 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 02:11:19.651957 23077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:19.653316 23063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:11:19.653475 23063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b2e5237958d44a7486250a897a242ce6"
format_stamp: "Formatted at 2024-05-03 02:11:19 on dist-test-slave-sc15"
I20240503 02:11:19.653661 23063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:19.669624 23063 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:19.670430 23063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:19.670703 23063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:19.671561 23063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:19.672920 23063 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:19.672997 23063 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:19.673146 23063 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:19.673199 23063 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:19.704568 23063 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.1:38179
I20240503 02:11:19.704609 23189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.1:38179 every 8 connection(s)
I20240503 02:11:19.705855 23063 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 02:11:19.712787 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 23063
I20240503 02:11:19.713016 22960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 02:11:19.716300 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.2:0
--local_ip_for_outbound_sockets=127.22.108.2
--webserver_interface=127.22.108.2
--webserver_port=0
--tserver_master_addrs=127.22.108.62:37253
--builtin_ntp_servers=127.22.108.20:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 02:11:19.718676 23190 heartbeater.cc:347] Connected to a master server at 127.22.108.62:37253
I20240503 02:11:19.718955 23190 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:19.719772 23190 heartbeater.cc:511] Master 127.22.108.62:37253 requested a full tablet report, sending...
I20240503 02:11:19.721925 23003 ts_manager.cc:194] Registered new tserver with Master: b2e5237958d44a7486250a897a242ce6 (127.22.108.1:38179)
I20240503 02:11:19.723364 23003 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.1:45461
W20240503 02:11:20.036304 23194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:20.036613 23194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:20.036763 23194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:20.044131 23194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:20.044373 23194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.2
I20240503 02:11:20.053459 23194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.108.2
--webserver_port=0
--tserver_master_addrs=127.22.108.62:37253
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.2
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:20.054813 23194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:20.056545 23194 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:20.068290 23199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:20.069067 23202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:20.069960 23200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:20.070161 23194 server_base.cc:1034] running on GCE node
I20240503 02:11:20.070803 23194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:20.071718 23194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:20.072933 23194 hybrid_clock.cc:648] HybridClock initialized: now 1714702280072878 us; error 30 us; skew 500 ppm
I20240503 02:11:20.073243 23194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:20.075495 23194 webserver.cc:460] Webserver started at http://127.22.108.2:39979/ using document root <none> and password file <none>
I20240503 02:11:20.076206 23194 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:20.076342 23194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:20.076628 23194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:20.079151 23194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "1dec96c923e64b43a2fbf2b681eb2f1d"
format_stamp: "Formatted at 2024-05-03 02:11:20 on dist-test-slave-sc15"
I20240503 02:11:20.079875 23194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "1dec96c923e64b43a2fbf2b681eb2f1d"
format_stamp: "Formatted at 2024-05-03 02:11:20 on dist-test-slave-sc15"
I20240503 02:11:20.084898 23194 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20240503 02:11:20.088536 23208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:20.089867 23194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:11:20.090025 23194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "1dec96c923e64b43a2fbf2b681eb2f1d"
format_stamp: "Formatted at 2024-05-03 02:11:20 on dist-test-slave-sc15"
I20240503 02:11:20.090195 23194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:20.119755 23194 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:20.120640 23194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:20.120946 23194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:20.121914 23194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:20.123385 23194 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:20.123466 23194 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:20.123627 23194 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:20.123684 23194 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:20.156231 23194 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.2:35435
I20240503 02:11:20.156265 23321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.2:35435 every 8 connection(s)
I20240503 02:11:20.157653 23194 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 02:11:20.163602 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 23194
I20240503 02:11:20.163770 22960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 02:11:20.167232 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.3:0
--local_ip_for_outbound_sockets=127.22.108.3
--webserver_interface=127.22.108.3
--webserver_port=0
--tserver_master_addrs=127.22.108.62:37253
--builtin_ntp_servers=127.22.108.20:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 02:11:20.170545 23322 heartbeater.cc:347] Connected to a master server at 127.22.108.62:37253
I20240503 02:11:20.170827 23322 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:20.171603 23322 heartbeater.cc:511] Master 127.22.108.62:37253 requested a full tablet report, sending...
I20240503 02:11:20.173136 23003 ts_manager.cc:194] Registered new tserver with Master: 1dec96c923e64b43a2fbf2b681eb2f1d (127.22.108.2:35435)
I20240503 02:11:20.174276 23003 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.2:38261
W20240503 02:11:20.502593 23325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:20.502888 23325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:20.502982 23325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:20.510602 23325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:20.510795 23325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.3
I20240503 02:11:20.519994 23325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.108.3
--webserver_port=0
--tserver_master_addrs=127.22.108.62:37253
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.3
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:20.521354 23325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:20.523046 23325 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:20.534644 23330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:20.535326 23333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:20.536168 23331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:20.536727 23325 server_base.cc:1034] running on GCE node
I20240503 02:11:20.537353 23325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:20.538255 23325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:20.539463 23325 hybrid_clock.cc:648] HybridClock initialized: now 1714702280539393 us; error 42 us; skew 500 ppm
I20240503 02:11:20.539760 23325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:20.542106 23325 webserver.cc:460] Webserver started at http://127.22.108.3:39671/ using document root <none> and password file <none>
I20240503 02:11:20.542826 23325 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:20.542946 23325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:20.543243 23325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:20.545846 23325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "a28b4393d0da4c7d850e55dfdf93bff7"
format_stamp: "Formatted at 2024-05-03 02:11:20 on dist-test-slave-sc15"
I20240503 02:11:20.546561 23325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "a28b4393d0da4c7d850e55dfdf93bff7"
format_stamp: "Formatted at 2024-05-03 02:11:20 on dist-test-slave-sc15"
I20240503 02:11:20.552522 23325 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20240503 02:11:20.556090 23339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:20.557610 23325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:11:20.557776 23325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "a28b4393d0da4c7d850e55dfdf93bff7"
format_stamp: "Formatted at 2024-05-03 02:11:20 on dist-test-slave-sc15"
I20240503 02:11:20.557991 23325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:20.586179 23325 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:20.587123 23325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:20.587427 23325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:20.588390 23325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:20.589876 23325 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:20.589956 23325 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:20.590099 23325 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:20.590155 23325 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:20.622254 23325 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.3:36661
I20240503 02:11:20.622323 23452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.3:36661 every 8 connection(s)
I20240503 02:11:20.623575 23325 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:11:20.625591 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 23325
I20240503 02:11:20.625797 22960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 02:11:20.629133 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.4:0
--local_ip_for_outbound_sockets=127.22.108.4
--webserver_interface=127.22.108.4
--webserver_port=0
--tserver_master_addrs=127.22.108.62:37253
--builtin_ntp_servers=127.22.108.20:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 02:11:20.636900 23453 heartbeater.cc:347] Connected to a master server at 127.22.108.62:37253
I20240503 02:11:20.637219 23453 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:20.638042 23453 heartbeater.cc:511] Master 127.22.108.62:37253 requested a full tablet report, sending...
I20240503 02:11:20.639514 23003 ts_manager.cc:194] Registered new tserver with Master: a28b4393d0da4c7d850e55dfdf93bff7 (127.22.108.3:36661)
I20240503 02:11:20.640305 23003 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.3:55829
I20240503 02:11:20.726135 23190 heartbeater.cc:503] Master 127.22.108.62:37253 was elected leader, sending a full tablet report...
W20240503 02:11:20.968026 23456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:20.968349 23456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:20.968506 23456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:20.976135 23456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:20.976374 23456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.4
I20240503 02:11:20.985821 23456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.108.4
--webserver_port=0
--tserver_master_addrs=127.22.108.62:37253
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.4
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:20.987313 23456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:20.989228 23456 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:21.001896 23461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:21.002470 23464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:21.003141 23462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:21.004019 23456 server_base.cc:1034] running on GCE node
I20240503 02:11:21.004669 23456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:21.005692 23456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:21.006919 23456 hybrid_clock.cc:648] HybridClock initialized: now 1714702281006857 us; error 40 us; skew 500 ppm
I20240503 02:11:21.007231 23456 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:21.009542 23456 webserver.cc:460] Webserver started at http://127.22.108.4:45195/ using document root <none> and password file <none>
I20240503 02:11:21.010335 23456 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:21.010473 23456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:21.010766 23456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:21.013468 23456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "61dfa54d8a0a4e16845a9d334be50525"
format_stamp: "Formatted at 2024-05-03 02:11:21 on dist-test-slave-sc15"
I20240503 02:11:21.014245 23456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "61dfa54d8a0a4e16845a9d334be50525"
format_stamp: "Formatted at 2024-05-03 02:11:21 on dist-test-slave-sc15"
I20240503 02:11:21.019660 23456 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 02:11:21.023113 23470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:21.024425 23456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 02:11:21.024585 23456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "61dfa54d8a0a4e16845a9d334be50525"
format_stamp: "Formatted at 2024-05-03 02:11:21 on dist-test-slave-sc15"
I20240503 02:11:21.024767 23456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:21.043824 23456 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:21.044682 23456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:21.044977 23456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:21.045917 23456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:21.047416 23456 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:21.047493 23456 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:21.047624 23456 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:21.047681 23456 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:21.079494 23456 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.4:39867
I20240503 02:11:21.079555 23582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.4:39867 every 8 connection(s)
I20240503 02:11:21.080822 23456 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:11:21.088260 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 23456
I20240503 02:11:21.088480 22960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 02:11:21.091665 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.5:0
--local_ip_for_outbound_sockets=127.22.108.5
--webserver_interface=127.22.108.5
--webserver_port=0
--tserver_master_addrs=127.22.108.62:37253
--builtin_ntp_servers=127.22.108.20:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 02:11:21.094012 23583 heartbeater.cc:347] Connected to a master server at 127.22.108.62:37253
I20240503 02:11:21.094301 23583 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:21.095176 23583 heartbeater.cc:511] Master 127.22.108.62:37253 requested a full tablet report, sending...
I20240503 02:11:21.096554 23003 ts_manager.cc:194] Registered new tserver with Master: 61dfa54d8a0a4e16845a9d334be50525 (127.22.108.4:39867)
I20240503 02:11:21.097385 23003 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.4:58309
I20240503 02:11:21.177286 23322 heartbeater.cc:503] Master 127.22.108.62:37253 was elected leader, sending a full tablet report...
W20240503 02:11:21.435405 23586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:21.435737 23586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:21.435894 23586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:21.443782 23586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:21.444010 23586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.5
I20240503 02:11:21.453495 23586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.108.5
--webserver_port=0
--tserver_master_addrs=127.22.108.62:37253
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.5
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:21.454888 23586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:21.456662 23586 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:21.469101 23591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:21.469841 23592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:21.470525 23594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:21.470875 23586 server_base.cc:1034] running on GCE node
I20240503 02:11:21.471506 23586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:21.472455 23586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:21.473685 23586 hybrid_clock.cc:648] HybridClock initialized: now 1714702281473609 us; error 51 us; skew 500 ppm
I20240503 02:11:21.473994 23586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:21.476243 23586 webserver.cc:460] Webserver started at http://127.22.108.5:39643/ using document root <none> and password file <none>
I20240503 02:11:21.476975 23586 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:21.477123 23586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:21.477430 23586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:21.480073 23586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "361f94e6ef4d4f7c8cbdc63c36f36f77"
format_stamp: "Formatted at 2024-05-03 02:11:21 on dist-test-slave-sc15"
I20240503 02:11:21.480816 23586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "361f94e6ef4d4f7c8cbdc63c36f36f77"
format_stamp: "Formatted at 2024-05-03 02:11:21 on dist-test-slave-sc15"
I20240503 02:11:21.485922 23586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20240503 02:11:21.489454 23600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:21.490749 23586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240503 02:11:21.490896 23586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "361f94e6ef4d4f7c8cbdc63c36f36f77"
format_stamp: "Formatted at 2024-05-03 02:11:21 on dist-test-slave-sc15"
I20240503 02:11:21.491075 23586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:21.514237 23586 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:21.515178 23586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:21.515471 23586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:21.516484 23586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:21.518070 23586 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:21.518146 23586 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:21.518255 23586 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:21.518316 23586 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:21.555747 23586 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.5:43337
I20240503 02:11:21.555788 23713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.5:43337 every 8 connection(s)
I20240503 02:11:21.557205 23586 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:11:21.560468 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 23586
I20240503 02:11:21.560645 22960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 02:11:21.569703 23714 heartbeater.cc:347] Connected to a master server at 127.22.108.62:37253
I20240503 02:11:21.569993 23714 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:21.571238 23714 heartbeater.cc:511] Master 127.22.108.62:37253 requested a full tablet report, sending...
I20240503 02:11:21.572561 23003 ts_manager.cc:194] Registered new tserver with Master: 361f94e6ef4d4f7c8cbdc63c36f36f77 (127.22.108.5:43337)
I20240503 02:11:21.573379 23003 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.5:58909
I20240503 02:11:21.576959 22960 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 02:11:21.598654 23003 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35652:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 02:11:21.642959 23453 heartbeater.cc:503] Master 127.22.108.62:37253 was elected leader, sending a full tablet report...
I20240503 02:11:21.647289 23388 tablet_service.cc:1467] Processing CreateTablet for tablet 146c143231bd453facbc4810a0d956d9 (DEFAULT_TABLE table=TestTable [id=888ed2f5dbca45a0af715d95bc8b4734]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:21.648797 23388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 146c143231bd453facbc4810a0d956d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:21.649629 23649 tablet_service.cc:1467] Processing CreateTablet for tablet 146c143231bd453facbc4810a0d956d9 (DEFAULT_TABLE table=TestTable [id=888ed2f5dbca45a0af715d95bc8b4734]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:21.649629 23518 tablet_service.cc:1467] Processing CreateTablet for tablet 146c143231bd453facbc4810a0d956d9 (DEFAULT_TABLE table=TestTable [id=888ed2f5dbca45a0af715d95bc8b4734]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:21.651069 23649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 146c143231bd453facbc4810a0d956d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:21.651068 23518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 146c143231bd453facbc4810a0d956d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:21.659559 23732 tablet_bootstrap.cc:492] T 146c143231bd453facbc4810a0d956d9 P a28b4393d0da4c7d850e55dfdf93bff7: Bootstrap starting.
I20240503 02:11:21.663146 23732 tablet_bootstrap.cc:654] T 146c143231bd453facbc4810a0d956d9 P a28b4393d0da4c7d850e55dfdf93bff7: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:21.664361 23732 log.cc:824] T 146c143231bd453facbc4810a0d956d9 P a28b4393d0da4c7d850e55dfdf93bff7: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:21.665864 23733 tablet_bootstrap.cc:492] T 146c143231bd453facbc4810a0d956d9 P 61dfa54d8a0a4e16845a9d334be50525: Bootstrap starting.
I20240503 02:11:21.666810 23734 tablet_bootstrap.cc:492] T 146c143231bd453facbc4810a0d956d9 P 361f94e6ef4d4f7c8cbdc63c36f36f77: Bootstrap starting.
I20240503 02:11:21.667943 23732 tablet_bootstrap.cc:492] T 146c143231bd453facbc4810a0d956d9 P a28b4393d0da4c7d850e55dfdf93bff7: No bootstrap required, opened a new log
I20240503 02:11:21.668233 23732 ts_tablet_manager.cc:1367] T 146c143231bd453facbc4810a0d956d9 P a28b4393d0da4c7d850e55dfdf93bff7: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240503 02:11:21.670336 23734 tablet_bootstrap.cc:654] T 146c143231bd453facbc4810a0d956d9 P 361f94e6ef4d4f7c8cbdc63c36f36f77: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:21.671854 23733 tablet_bootstrap.cc:654] T 146c143231bd453facbc4810a0d956d9 P 61dfa54d8a0a4e16845a9d334be50525: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:21.671916 23734 log.cc:824] T 146c143231bd453facbc4810a0d956d9 P 361f94e6ef4d4f7c8cbdc63c36f36f77: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:21.672999 23733 log.cc:824] T 146c143231bd453facbc4810a0d956d9 P 61dfa54d8a0a4e16845a9d334be50525: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:21.674983 23734 tablet_bootstrap.cc:492] T 146c143231bd453facbc4810a0d956d9 P 361f94e6ef4d4f7c8cbdc63c36f36f77: No bootstrap required, opened a new log
I20240503 02:11:21.675251 23734 ts_tablet_manager.cc:1367] T 146c143231bd453facbc4810a0d956d9 P 361f94e6ef4d4f7c8cbdc63c36f36f77: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240503 02:11:21.675599 23733 tablet_bootstrap.cc:492] T 146c143231bd453facbc4810a0d956d9 P 61dfa54d8a0a4e16845a9d334be50525: No bootstrap required, opened a new log
I20240503 02:11:21.675841 23733 ts_tablet_manager.cc:1367] T 146c143231bd453facbc4810a0d956d9 P 61dfa54d8a0a4e16845a9d334be50525: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20240503 02:11:21.677253 23732 raft_consensus.cc:348] T 146c143231bd453facbc4810a0d956d9 P a28b4393d0da4c7d850e55dfdf93bff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28b4393d0da4c7d850e55dfdf93bff7" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 36661 } } peers { permanent_uuid: "61dfa54d8a0a4e16845a9d334be50525" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39867 } } peers { permanent_uuid: "361f94e6ef4d4f7c8cbdc63c36f36f77" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 43337 } }
I20240503 02:11:21.677716 23732 raft_consensus.cc:374] T 146c143231bd453facbc4810a0d956d9 P a28b4393d0da4c7d850e55dfdf93bff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:21.677821 23732 raft_consensus.cc:729] T 146c143231bd453facbc4810a0d956d9 P a28b4393d0da4c7d850e55dfdf93bff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a28b4393d0da4c7d850e55dfdf93bff7, State: Initialized, Role: FOLLOWER
I20240503 02:11:21.678526 23732 consensus_queue.cc:260] T 146c143231bd453facbc4810a0d956d9 P a28b4393d0da4c7d850e55dfdf93bff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28b4393d0da4c7d850e55dfdf93bff7" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 36661 } } peers { permanent_uuid: "61dfa54d8a0a4e16845a9d334be50525" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39867 } } peers { permanent_uuid: "361f94e6ef4d4f7c8cbdc63c36f36f77" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 43337 } }
I20240503 02:11:21.680311 23732 ts_tablet_manager.cc:1398] T 146c143231bd453facbc4810a0d956d9 P a28b4393d0da4c7d850e55dfdf93bff7: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20240503 02:11:21.683053 23733 raft_consensus.cc:348] T 146c143231bd453facbc4810a0d956d9 P 61dfa54d8a0a4e16845a9d334be50525 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28b4393d0da4c7d850e55dfdf93bff7" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 36661 } } peers { permanent_uuid: "61dfa54d8a0a4e16845a9d334be50525" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39867 } } peers { permanent_uuid: "361f94e6ef4d4f7c8cbdc63c36f36f77" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 43337 } }
I20240503 02:11:21.683406 23733 raft_consensus.cc:374] T 146c143231bd453facbc4810a0d956d9 P 61dfa54d8a0a4e16845a9d334be50525 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:21.683522 23733 raft_consensus.cc:729] T 146c143231bd453facbc4810a0d956d9 P 61dfa54d8a0a4e16845a9d334be50525 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61dfa54d8a0a4e16845a9d334be50525, State: Initialized, Role: FOLLOWER
I20240503 02:11:21.684248 23733 consensus_queue.cc:260] T 146c143231bd453facbc4810a0d956d9 P 61dfa54d8a0a4e16845a9d334be50525 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28b4393d0da4c7d850e55dfdf93bff7" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 36661 } } peers { permanent_uuid: "61dfa54d8a0a4e16845a9d334be50525" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39867 } } peers { permanent_uuid: "361f94e6ef4d4f7c8cbdc63c36f36f77" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 43337 } }
I20240503 02:11:21.684275 23734 raft_consensus.cc:348] T 146c143231bd453facbc4810a0d956d9 P 361f94e6ef4d4f7c8cbdc63c36f36f77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28b4393d0da4c7d850e55dfdf93bff7" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 36661 } } peers { permanent_uuid: "61dfa54d8a0a4e16845a9d334be50525" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39867 } } peers { permanent_uuid: "361f94e6ef4d4f7c8cbdc63c36f36f77" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 43337 } }
I20240503 02:11:21.684679 23734 raft_consensus.cc:374] T 146c143231bd453facbc4810a0d956d9 P 361f94e6ef4d4f7c8cbdc63c36f36f77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:21.684777 23734 raft_consensus.cc:729] T 146c143231bd453facbc4810a0d956d9 P 361f94e6ef4d4f7c8cbdc63c36f36f77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 361f94e6ef4d4f7c8cbdc63c36f36f77, State: Initialized, Role: FOLLOWER
I20240503 02:11:21.685529 23734 consensus_queue.cc:260] T 146c143231bd453facbc4810a0d956d9 P 361f94e6ef4d4f7c8cbdc63c36f36f77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28b4393d0da4c7d850e55dfdf93bff7" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 36661 } } peers { permanent_uuid: "61dfa54d8a0a4e16845a9d334be50525" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39867 } } peers { permanent_uuid: "361f94e6ef4d4f7c8cbdc63c36f36f77" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 43337 } }
I20240503 02:11:21.685727 23583 heartbeater.cc:503] Master 127.22.108.62:37253 was elected leader, sending a full tablet report...
I20240503 02:11:21.686568 23714 heartbeater.cc:503] Master 127.22.108.62:37253 was elected leader, sending a full tablet report...
I20240503 02:11:21.687258 23734 ts_tablet_manager.cc:1398] T 146c143231bd453facbc4810a0d956d9 P 361f94e6ef4d4f7c8cbdc63c36f36f77: Time spent starting tablet: real 0.012s user 0.009s sys 0.005s
I20240503 02:11:21.687830 23733 ts_tablet_manager.cc:1398] T 146c143231bd453facbc4810a0d956d9 P 61dfa54d8a0a4e16845a9d334be50525: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
W20240503 02:11:21.808862 23715 tablet.cc:2369] T 146c143231bd453facbc4810a0d956d9 P 361f94e6ef4d4f7c8cbdc63c36f36f77: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:11:21.832810 23584 tablet.cc:2369] T 146c143231bd453facbc4810a0d956d9 P 61dfa54d8a0a4e16845a9d334be50525: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:11:21.876044 23454 tablet.cc:2369] T 146c143231bd453facbc4810a0d956d9 P a28b4393d0da4c7d850e55dfdf93bff7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:11:22.034405 23740 raft_consensus.cc:482] T 146c143231bd453facbc4810a0d956d9 P 361f94e6ef4d4f7c8cbdc63c36f36f77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:22.034690 23740 raft_consensus.cc:504] T 146c143231bd453facbc4810a0d956d9 P 361f94e6ef4d4f7c8cbdc63c36f36f77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28b4393d0da4c7d850e55dfdf93bff7" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 36661 } } peers { permanent_uuid: "61dfa54d8a0a4e16845a9d334be50525" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39867 } } peers { permanent_uuid: "361f94e6ef4d4f7c8cbdc63c36f36f77" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 43337 } }
I20240503 02:11:22.036819 23740 leader_election.cc:288] T 146c143231bd453facbc4810a0d956d9 P 361f94e6ef4d4f7c8cbdc63c36f36f77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a28b4393d0da4c7d850e55dfdf93bff7 (127.22.108.3:36661), 61dfa54d8a0a4e16845a9d334be50525 (127.22.108.4:39867)
I20240503 02:11:22.043602 23408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "146c143231bd453facbc4810a0d956d9" candidate_uuid: "361f94e6ef4d4f7c8cbdc63c36f36f77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a28b4393d0da4c7d850e55dfdf93bff7" is_pre_election: true
I20240503 02:11:22.043717 23538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "146c143231bd453facbc4810a0d956d9" candidate_uuid: "361f94e6ef4d4f7c8cbdc63c36f36f77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61dfa54d8a0a4e16845a9d334be50525" is_pre_election: true
I20240503 02:11:22.044111 23408 raft_consensus.cc:2454] T 146c143231bd453facbc4810a0d956d9 P a28b4393d0da4c7d850e55dfdf93bff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 361f94e6ef4d4f7c8cbdc63c36f36f77 in term 0.
I20240503 02:11:22.044188 23538 raft_consensus.cc:2454] T 146c143231bd453facbc4810a0d956d9 P 61dfa54d8a0a4e16845a9d334be50525 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 361f94e6ef4d4f7c8cbdc63c36f36f77 in term 0.
I20240503 02:11:22.044771 23603 leader_election.cc:302] T 146c143231bd453facbc4810a0d956d9 P 361f94e6ef4d4f7c8cbdc63c36f36f77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 361f94e6ef4d4f7c8cbdc63c36f36f77, a28b4393d0da4c7d850e55dfdf93bff7; no voters:
I20240503 02:11:22.045159 23740 raft_consensus.cc:2781] T 146c143231bd453facbc4810a0d956d9 P 361f94e6ef4d4f7c8cbdc63c36f36f77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:22.045325 23740 raft_consensus.cc:482] T 146c143231bd453facbc4810a0d956d9 P 361f94e6ef4d4f7c8cbdc63c36f36f77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:22.045428 23740 raft_consensus.cc:3037] T 146c143231bd453facbc4810a0d956d9 P 361f94e6ef4d4f7c8cbdc63c36f36f77 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:22.047394 23740 raft_consensus.cc:504] T 146c143231bd453facbc4810a0d956d9 P 361f94e6ef4d4f7c8cbdc63c36f36f77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28b4393d0da4c7d850e55dfdf93bff7" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 36661 } } peers { permanent_uuid: "61dfa54d8a0a4e16845a9d334be50525" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39867 } } peers { permanent_uuid: "361f94e6ef4d4f7c8cbdc63c36f36f77" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 43337 } }
I20240503 02:11:22.047971 23740 leader_election.cc:288] T 146c143231bd453facbc4810a0d956d9 P 361f94e6ef4d4f7c8cbdc63c36f36f77 [CANDIDATE]: Term 1 election: Requested vote from peers a28b4393d0da4c7d850e55dfdf93bff7 (127.22.108.3:36661), 61dfa54d8a0a4e16845a9d334be50525 (127.22.108.4:39867)
I20240503 02:11:22.048439 23408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "146c143231bd453facbc4810a0d956d9" candidate_uuid: "361f94e6ef4d4f7c8cbdc63c36f36f77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a28b4393d0da4c7d850e55dfdf93bff7"
I20240503 02:11:22.048462 23538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "146c143231bd453facbc4810a0d956d9" candidate_uuid: "361f94e6ef4d4f7c8cbdc63c36f36f77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61dfa54d8a0a4e16845a9d334be50525"
I20240503 02:11:22.048656 23538 raft_consensus.cc:3037] T 146c143231bd453facbc4810a0d956d9 P 61dfa54d8a0a4e16845a9d334be50525 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:22.048656 23408 raft_consensus.cc:3037] T 146c143231bd453facbc4810a0d956d9 P a28b4393d0da4c7d850e55dfdf93bff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:22.051757 23538 raft_consensus.cc:2454] T 146c143231bd453facbc4810a0d956d9 P 61dfa54d8a0a4e16845a9d334be50525 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 361f94e6ef4d4f7c8cbdc63c36f36f77 in term 1.
I20240503 02:11:22.051757 23408 raft_consensus.cc:2454] T 146c143231bd453facbc4810a0d956d9 P a28b4393d0da4c7d850e55dfdf93bff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 361f94e6ef4d4f7c8cbdc63c36f36f77 in term 1.
I20240503 02:11:22.052189 23603 leader_election.cc:302] T 146c143231bd453facbc4810a0d956d9 P 361f94e6ef4d4f7c8cbdc63c36f36f77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 361f94e6ef4d4f7c8cbdc63c36f36f77, a28b4393d0da4c7d850e55dfdf93bff7; no voters:
I20240503 02:11:22.052482 23740 raft_consensus.cc:2781] T 146c143231bd453facbc4810a0d956d9 P 361f94e6ef4d4f7c8cbdc63c36f36f77 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:22.052901 23740 raft_consensus.cc:686] T 146c143231bd453facbc4810a0d956d9 P 361f94e6ef4d4f7c8cbdc63c36f36f77 [term 1 LEADER]: Becoming Leader. State: Replica: 361f94e6ef4d4f7c8cbdc63c36f36f77, State: Running, Role: LEADER
I20240503 02:11:22.053455 23740 consensus_queue.cc:237] T 146c143231bd453facbc4810a0d956d9 P 361f94e6ef4d4f7c8cbdc63c36f36f77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28b4393d0da4c7d850e55dfdf93bff7" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 36661 } } peers { permanent_uuid: "61dfa54d8a0a4e16845a9d334be50525" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39867 } } peers { permanent_uuid: "361f94e6ef4d4f7c8cbdc63c36f36f77" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 43337 } }
I20240503 02:11:22.058043 23003 catalog_manager.cc:5484] T 146c143231bd453facbc4810a0d956d9 P 361f94e6ef4d4f7c8cbdc63c36f36f77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 361f94e6ef4d4f7c8cbdc63c36f36f77 (127.22.108.5). New cstate: current_term: 1 leader_uuid: "361f94e6ef4d4f7c8cbdc63c36f36f77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28b4393d0da4c7d850e55dfdf93bff7" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 36661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61dfa54d8a0a4e16845a9d334be50525" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "361f94e6ef4d4f7c8cbdc63c36f36f77" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 43337 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:22.096076 22960 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 02:11:22.098731 22960 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b2e5237958d44a7486250a897a242ce6 to finish bootstrapping
I20240503 02:11:22.107601 22960 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 1dec96c923e64b43a2fbf2b681eb2f1d to finish bootstrapping
I20240503 02:11:22.117103 22960 ts_itest-base.cc:233] Waiting for 1 tablets on tserver a28b4393d0da4c7d850e55dfdf93bff7 to finish bootstrapping
I20240503 02:11:22.124852 22960 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 61dfa54d8a0a4e16845a9d334be50525 to finish bootstrapping
I20240503 02:11:22.133102 22960 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 361f94e6ef4d4f7c8cbdc63c36f36f77 to finish bootstrapping
I20240503 02:11:22.534979 23740 consensus_queue.cc:1035] T 146c143231bd453facbc4810a0d956d9 P 361f94e6ef4d4f7c8cbdc63c36f36f77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a28b4393d0da4c7d850e55dfdf93bff7" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 36661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:22.544275 23740 consensus_queue.cc:1035] T 146c143231bd453facbc4810a0d956d9 P 361f94e6ef4d4f7c8cbdc63c36f36f77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61dfa54d8a0a4e16845a9d334be50525" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:22.646301 23257 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:22.647348 23649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:22.667505 23518 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:22.687242 23125 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:22.699162 23388 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:22.844321 23257 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:22.846881 23649 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:22.853575 23518 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:22.860832 23125 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:22.862249 23388 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:23.005712 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 23063
I20240503 02:11:23.007092 23184 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:23.090605 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 23063
I20240503 02:11:23.113529 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 23194
I20240503 02:11:23.114519 23316 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:23.204519 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 23194
I20240503 02:11:23.221189 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 23325
I20240503 02:11:23.222151 23448 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:23.308445 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 23325
I20240503 02:11:23.325424 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 23456
I20240503 02:11:23.326438 23578 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:23.412760 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 23456
I20240503 02:11:23.429910 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 23586
I20240503 02:11:23.430886 23709 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:23.519356 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 23586
I20240503 02:11:23.536456 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 22972
I20240503 02:11:23.537490 23033 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:23.644279 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 22972
2024-05-03T02:11:23Z chronyd exiting
[ OK ] AdminCliTest.RebalancerReportOnly (4985 ms)
[----------] 1 test from AdminCliTest (4985 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.TabletServerIsDown/0
I20240503 02:11:23.686065 22960 test_util.cc:268] Using random seed: 1015267466
I20240503 02:11:23.687752 22960 ts_itest-base.cc:112] Starting cluster with:
I20240503 02:11:23.687835 22960 ts_itest-base.cc:113] --------------
I20240503 02:11:23.687891 22960 ts_itest-base.cc:114] 5 tablet servers
I20240503 02:11:23.687956 22960 ts_itest-base.cc:115] 3 replicas per TS
I20240503 02:11:23.688006 22960 ts_itest-base.cc:116] --------------
2024-05-03T02:11:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:11:23Z Disabled control of system clock
I20240503 02:11:23.710163 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.108.62:43769
--webserver_interface=127.22.108.62
--webserver_port=0
--builtin_ntp_servers=127.22.108.20:35691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.108.62:43769
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240503 02:11:24.018955 23838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:24.019279 23838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:24.019447 23838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:24.026698 23838 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:11:24.026834 23838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:11:24.026906 23838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:24.026993 23838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:11:24.027060 23838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:11:24.035584 23838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:35691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.22.108.62:43769
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.108.62:43769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.108.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:24.036826 23838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:24.038558 23838 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:24.049153 23843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:24.049885 23846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:24.050575 23844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:24.051221 23838 server_base.cc:1034] running on GCE node
I20240503 02:11:24.051947 23838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:24.053361 23838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:24.054566 23838 hybrid_clock.cc:648] HybridClock initialized: now 1714702284054527 us; error 33 us; skew 500 ppm
I20240503 02:11:24.054898 23838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:24.062587 23838 webserver.cc:460] Webserver started at http://127.22.108.62:39513/ using document root <none> and password file <none>
I20240503 02:11:24.063302 23838 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:24.063429 23838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:24.063758 23838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:24.066303 23838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "02efccf2d25a4e7f93b5accf0c915fa5"
format_stamp: "Formatted at 2024-05-03 02:11:24 on dist-test-slave-sc15"
I20240503 02:11:24.067027 23838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "02efccf2d25a4e7f93b5accf0c915fa5"
format_stamp: "Formatted at 2024-05-03 02:11:24 on dist-test-slave-sc15"
I20240503 02:11:24.072002 23838 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 02:11:24.075258 23852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:24.076475 23838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 02:11:24.076633 23838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/wal
uuid: "02efccf2d25a4e7f93b5accf0c915fa5"
format_stamp: "Formatted at 2024-05-03 02:11:24 on dist-test-slave-sc15"
I20240503 02:11:24.076813 23838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:24.096366 23838 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:24.097178 23838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:24.097443 23838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:24.114511 23838 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.62:43769
I20240503 02:11:24.114550 23903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.62:43769 every 8 connection(s)
I20240503 02:11:24.115779 23838 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 02:11:24.119249 23904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:24.125759 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 23838
I20240503 02:11:24.125981 22960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 02:11:24.128778 23904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02efccf2d25a4e7f93b5accf0c915fa5: Bootstrap starting.
I20240503 02:11:24.131830 23904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02efccf2d25a4e7f93b5accf0c915fa5: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:24.133100 23904 log.cc:824] T 00000000000000000000000000000000 P 02efccf2d25a4e7f93b5accf0c915fa5: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:24.135666 23904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02efccf2d25a4e7f93b5accf0c915fa5: No bootstrap required, opened a new log
I20240503 02:11:24.142446 23904 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02efccf2d25a4e7f93b5accf0c915fa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02efccf2d25a4e7f93b5accf0c915fa5" member_type: VOTER last_known_addr { host: "127.22.108.62" port: 43769 } }
I20240503 02:11:24.142742 23904 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02efccf2d25a4e7f93b5accf0c915fa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:24.142851 23904 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02efccf2d25a4e7f93b5accf0c915fa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02efccf2d25a4e7f93b5accf0c915fa5, State: Initialized, Role: FOLLOWER
I20240503 02:11:24.143467 23904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02efccf2d25a4e7f93b5accf0c915fa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02efccf2d25a4e7f93b5accf0c915fa5" member_type: VOTER last_known_addr { host: "127.22.108.62" port: 43769 } }
I20240503 02:11:24.143661 23904 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02efccf2d25a4e7f93b5accf0c915fa5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:24.143754 23904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02efccf2d25a4e7f93b5accf0c915fa5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:24.143864 23904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02efccf2d25a4e7f93b5accf0c915fa5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:24.146003 23904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02efccf2d25a4e7f93b5accf0c915fa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02efccf2d25a4e7f93b5accf0c915fa5" member_type: VOTER last_known_addr { host: "127.22.108.62" port: 43769 } }
I20240503 02:11:24.146513 23904 leader_election.cc:302] T 00000000000000000000000000000000 P 02efccf2d25a4e7f93b5accf0c915fa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02efccf2d25a4e7f93b5accf0c915fa5; no voters:
I20240503 02:11:24.147006 23904 leader_election.cc:288] T 00000000000000000000000000000000 P 02efccf2d25a4e7f93b5accf0c915fa5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:24.147135 23909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02efccf2d25a4e7f93b5accf0c915fa5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:24.147712 23909 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02efccf2d25a4e7f93b5accf0c915fa5 [term 1 LEADER]: Becoming Leader. State: Replica: 02efccf2d25a4e7f93b5accf0c915fa5, State: Running, Role: LEADER
I20240503 02:11:24.148201 23909 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02efccf2d25a4e7f93b5accf0c915fa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02efccf2d25a4e7f93b5accf0c915fa5" member_type: VOTER last_known_addr { host: "127.22.108.62" port: 43769 } }
I20240503 02:11:24.148737 23904 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02efccf2d25a4e7f93b5accf0c915fa5 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:11:24.150931 23910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02efccf2d25a4e7f93b5accf0c915fa5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02efccf2d25a4e7f93b5accf0c915fa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02efccf2d25a4e7f93b5accf0c915fa5" member_type: VOTER last_known_addr { host: "127.22.108.62" port: 43769 } } }
I20240503 02:11:24.150990 23911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02efccf2d25a4e7f93b5accf0c915fa5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02efccf2d25a4e7f93b5accf0c915fa5. Latest consensus state: current_term: 1 leader_uuid: "02efccf2d25a4e7f93b5accf0c915fa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02efccf2d25a4e7f93b5accf0c915fa5" member_type: VOTER last_known_addr { host: "127.22.108.62" port: 43769 } } }
I20240503 02:11:24.151372 23910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02efccf2d25a4e7f93b5accf0c915fa5 [sys.catalog]: This master's current role is: LEADER
I20240503 02:11:24.151479 23911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02efccf2d25a4e7f93b5accf0c915fa5 [sys.catalog]: This master's current role is: LEADER
I20240503 02:11:24.152910 23917 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:11:24.159006 23917 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:11:24.167249 23917 catalog_manager.cc:1347] Generated new cluster ID: 12dec3a53e4c4ad98205199aeec4aaa2
I20240503 02:11:24.167372 23917 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:11:24.183847 23917 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:11:24.185073 23917 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:11:24.194339 23917 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 02efccf2d25a4e7f93b5accf0c915fa5: Generated new TSK 0
I20240503 02:11:24.195104 23917 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:11:24.202685 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.1:0
--local_ip_for_outbound_sockets=127.22.108.1
--webserver_interface=127.22.108.1
--webserver_port=0
--tserver_master_addrs=127.22.108.62:43769
--builtin_ntp_servers=127.22.108.20:35691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 02:11:24.521709 23928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:24.522020 23928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:24.522152 23928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:24.529512 23928 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:11:24.529621 23928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:24.529774 23928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.1
I20240503 02:11:24.538491 23928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:35691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.108.1
--webserver_port=0
--tserver_master_addrs=127.22.108.62:43769
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.1
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:24.539801 23928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:24.541606 23928 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:24.553076 23933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:24.553733 23936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:24.554416 23934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:24.555212 23928 server_base.cc:1034] running on GCE node
I20240503 02:11:24.555770 23928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:24.556634 23928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:24.557823 23928 hybrid_clock.cc:648] HybridClock initialized: now 1714702284557770 us; error 44 us; skew 500 ppm
I20240503 02:11:24.558107 23928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:24.560163 23928 webserver.cc:460] Webserver started at http://127.22.108.1:33463/ using document root <none> and password file <none>
I20240503 02:11:24.560875 23928 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:24.560981 23928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:24.561302 23928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:24.564052 23928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "d60d3644181e48818a66372f8a0c9ed7"
format_stamp: "Formatted at 2024-05-03 02:11:24 on dist-test-slave-sc15"
I20240503 02:11:24.565066 23928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "d60d3644181e48818a66372f8a0c9ed7"
format_stamp: "Formatted at 2024-05-03 02:11:24 on dist-test-slave-sc15"
I20240503 02:11:24.570277 23928 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240503 02:11:24.573544 23942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:24.574836 23928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20240503 02:11:24.574985 23928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d60d3644181e48818a66372f8a0c9ed7"
format_stamp: "Formatted at 2024-05-03 02:11:24 on dist-test-slave-sc15"
I20240503 02:11:24.575165 23928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:24.588173 23928 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:24.588891 23928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:24.589170 23928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:24.590058 23928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:24.591426 23928 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:24.591531 23928 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:24.591640 23928 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:24.591709 23928 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:24.621434 23928 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.1:36335
I20240503 02:11:24.621526 24054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.1:36335 every 8 connection(s)
I20240503 02:11:24.622819 23928 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 02:11:24.630722 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 23928
I20240503 02:11:24.630928 22960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 02:11:24.634625 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.2:0
--local_ip_for_outbound_sockets=127.22.108.2
--webserver_interface=127.22.108.2
--webserver_port=0
--tserver_master_addrs=127.22.108.62:43769
--builtin_ntp_servers=127.22.108.20:35691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:11:24.635107 24055 heartbeater.cc:347] Connected to a master server at 127.22.108.62:43769
I20240503 02:11:24.635380 24055 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:24.636049 24055 heartbeater.cc:511] Master 127.22.108.62:43769 requested a full tablet report, sending...
I20240503 02:11:24.638062 23869 ts_manager.cc:194] Registered new tserver with Master: d60d3644181e48818a66372f8a0c9ed7 (127.22.108.1:36335)
I20240503 02:11:24.639726 23869 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.1:47919
W20240503 02:11:24.961427 24058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:24.961750 24058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:24.961905 24058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:24.969399 24058 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:11:24.969580 24058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:24.969745 24058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.2
I20240503 02:11:24.978950 24058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:35691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.108.2
--webserver_port=0
--tserver_master_addrs=127.22.108.62:43769
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.2
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:24.980396 24058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:24.982276 24058 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:24.994383 24063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:24.995249 24064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:24.995764 24066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:24.996619 24058 server_base.cc:1034] running on GCE node
I20240503 02:11:24.997259 24058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:24.998214 24058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:24.999413 24058 hybrid_clock.cc:648] HybridClock initialized: now 1714702284999369 us; error 39 us; skew 500 ppm
I20240503 02:11:24.999737 24058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:25.002095 24058 webserver.cc:460] Webserver started at http://127.22.108.2:39221/ using document root <none> and password file <none>
I20240503 02:11:25.002843 24058 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:25.002964 24058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:25.003285 24058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:25.006247 24058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "cacd0decd933496798fe28a7a793f947"
format_stamp: "Formatted at 2024-05-03 02:11:24 on dist-test-slave-sc15"
I20240503 02:11:25.007254 24058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "cacd0decd933496798fe28a7a793f947"
format_stamp: "Formatted at 2024-05-03 02:11:24 on dist-test-slave-sc15"
I20240503 02:11:25.012610 24058 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 02:11:25.016232 24073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:25.017671 24058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240503 02:11:25.017843 24058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "cacd0decd933496798fe28a7a793f947"
format_stamp: "Formatted at 2024-05-03 02:11:24 on dist-test-slave-sc15"
I20240503 02:11:25.018049 24058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:25.039072 24058 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:25.039985 24058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:25.040272 24058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:25.041402 24058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:25.043027 24058 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:25.043133 24058 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:25.043256 24058 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:25.043339 24058 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:25.077718 24058 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.2:38073
I20240503 02:11:25.077798 24185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.2:38073 every 8 connection(s)
I20240503 02:11:25.079190 24058 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 02:11:25.082512 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 24058
I20240503 02:11:25.082734 22960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 02:11:25.086786 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.3:0
--local_ip_for_outbound_sockets=127.22.108.3
--webserver_interface=127.22.108.3
--webserver_port=0
--tserver_master_addrs=127.22.108.62:43769
--builtin_ntp_servers=127.22.108.20:35691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:11:25.095213 24186 heartbeater.cc:347] Connected to a master server at 127.22.108.62:43769
I20240503 02:11:25.095546 24186 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:25.096334 24186 heartbeater.cc:511] Master 127.22.108.62:43769 requested a full tablet report, sending...
I20240503 02:11:25.097877 23869 ts_manager.cc:194] Registered new tserver with Master: cacd0decd933496798fe28a7a793f947 (127.22.108.2:38073)
I20240503 02:11:25.099108 23869 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.2:36269
W20240503 02:11:25.417016 24189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:25.417392 24189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:25.417495 24189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:25.425204 24189 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:11:25.425333 24189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:25.425467 24189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.3
I20240503 02:11:25.434620 24189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:35691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.108.3
--webserver_port=0
--tserver_master_addrs=127.22.108.62:43769
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.3
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:25.435931 24189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:25.437712 24189 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:25.449611 24194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:25.450419 24195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:25.450898 24197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:25.451800 24189 server_base.cc:1034] running on GCE node
I20240503 02:11:25.452394 24189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:25.453346 24189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:25.454541 24189 hybrid_clock.cc:648] HybridClock initialized: now 1714702285454493 us; error 45 us; skew 500 ppm
I20240503 02:11:25.454846 24189 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:25.457222 24189 webserver.cc:460] Webserver started at http://127.22.108.3:45201/ using document root <none> and password file <none>
I20240503 02:11:25.457975 24189 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:25.458091 24189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:25.458421 24189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:25.460990 24189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "b8aeb791035f4f04b304ef08b49dd7b6"
format_stamp: "Formatted at 2024-05-03 02:11:25 on dist-test-slave-sc15"
I20240503 02:11:25.461808 24189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "b8aeb791035f4f04b304ef08b49dd7b6"
format_stamp: "Formatted at 2024-05-03 02:11:25 on dist-test-slave-sc15"
I20240503 02:11:25.467053 24189 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240503 02:11:25.470530 24203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:25.471881 24189 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 02:11:25.472040 24189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b8aeb791035f4f04b304ef08b49dd7b6"
format_stamp: "Formatted at 2024-05-03 02:11:25 on dist-test-slave-sc15"
I20240503 02:11:25.472239 24189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:25.498595 24189 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:25.499493 24189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:25.499763 24189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:25.500720 24189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:25.502355 24189 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:25.502570 24189 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:25.502712 24189 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:25.502789 24189 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:25.534207 24189 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.3:33185
I20240503 02:11:25.534250 24315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.3:33185 every 8 connection(s)
I20240503 02:11:25.535583 24189 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:11:25.536072 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 24189
I20240503 02:11:25.536237 22960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 02:11:25.539646 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.4:0
--local_ip_for_outbound_sockets=127.22.108.4
--webserver_interface=127.22.108.4
--webserver_port=0
--tserver_master_addrs=127.22.108.62:43769
--builtin_ntp_servers=127.22.108.20:35691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:11:25.549366 24316 heartbeater.cc:347] Connected to a master server at 127.22.108.62:43769
I20240503 02:11:25.549721 24316 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:25.550563 24316 heartbeater.cc:511] Master 127.22.108.62:43769 requested a full tablet report, sending...
I20240503 02:11:25.551951 23869 ts_manager.cc:194] Registered new tserver with Master: b8aeb791035f4f04b304ef08b49dd7b6 (127.22.108.3:33185)
I20240503 02:11:25.552731 23869 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.3:46983
I20240503 02:11:25.642508 24055 heartbeater.cc:503] Master 127.22.108.62:43769 was elected leader, sending a full tablet report...
W20240503 02:11:25.873020 24319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:25.873476 24319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:25.873631 24319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:25.882102 24319 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:11:25.882225 24319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:25.882395 24319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.4
I20240503 02:11:25.891707 24319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:35691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.108.4
--webserver_port=0
--tserver_master_addrs=127.22.108.62:43769
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.4
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:25.893118 24319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:25.894922 24319 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:25.906688 24324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:25.907481 24325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:25.908175 24327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:25.909113 24319 server_base.cc:1034] running on GCE node
I20240503 02:11:25.909682 24319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:25.910555 24319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:25.911753 24319 hybrid_clock.cc:648] HybridClock initialized: now 1714702285911713 us; error 40 us; skew 500 ppm
I20240503 02:11:25.912055 24319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:25.914208 24319 webserver.cc:460] Webserver started at http://127.22.108.4:43635/ using document root <none> and password file <none>
I20240503 02:11:25.914911 24319 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:25.915033 24319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:25.915338 24319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:25.917879 24319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "4a36fb8d35744a5fb5505391524a2cf0"
format_stamp: "Formatted at 2024-05-03 02:11:25 on dist-test-slave-sc15"
I20240503 02:11:25.918628 24319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "4a36fb8d35744a5fb5505391524a2cf0"
format_stamp: "Formatted at 2024-05-03 02:11:25 on dist-test-slave-sc15"
I20240503 02:11:25.923682 24319 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 02:11:25.926957 24333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:25.928261 24319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 02:11:25.928421 24319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "4a36fb8d35744a5fb5505391524a2cf0"
format_stamp: "Formatted at 2024-05-03 02:11:25 on dist-test-slave-sc15"
I20240503 02:11:25.928639 24319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:25.944840 24319 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:25.945703 24319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:25.945968 24319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:25.946869 24319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:25.948365 24319 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:25.948480 24319 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:25.948592 24319 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:25.948678 24319 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:25.980530 24319 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.4:41491
I20240503 02:11:25.980580 24445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.4:41491 every 8 connection(s)
I20240503 02:11:25.981997 24319 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:11:25.988901 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 24319
I20240503 02:11:25.989122 22960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 02:11:25.992221 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.5:0
--local_ip_for_outbound_sockets=127.22.108.5
--webserver_interface=127.22.108.5
--webserver_port=0
--tserver_master_addrs=127.22.108.62:43769
--builtin_ntp_servers=127.22.108.20:35691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:11:25.994588 24446 heartbeater.cc:347] Connected to a master server at 127.22.108.62:43769
I20240503 02:11:25.994905 24446 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:25.995767 24446 heartbeater.cc:511] Master 127.22.108.62:43769 requested a full tablet report, sending...
I20240503 02:11:25.997249 23869 ts_manager.cc:194] Registered new tserver with Master: 4a36fb8d35744a5fb5505391524a2cf0 (127.22.108.4:41491)
I20240503 02:11:25.998031 23869 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.4:50331
I20240503 02:11:26.102087 24186 heartbeater.cc:503] Master 127.22.108.62:43769 was elected leader, sending a full tablet report...
W20240503 02:11:26.318218 24449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:26.318570 24449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:26.318724 24449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:26.326165 24449 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:11:26.326337 24449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:26.326515 24449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.5
I20240503 02:11:26.335623 24449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:35691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.108.5
--webserver_port=0
--tserver_master_addrs=127.22.108.62:43769
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.5
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:26.337000 24449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:26.338826 24449 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:26.351361 24457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:26.352270 24455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:26.352290 24454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:26.352981 24449 server_base.cc:1034] running on GCE node
I20240503 02:11:26.353610 24449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:26.354511 24449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:26.355708 24449 hybrid_clock.cc:648] HybridClock initialized: now 1714702286355647 us; error 54 us; skew 500 ppm
I20240503 02:11:26.356069 24449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:26.358403 24449 webserver.cc:460] Webserver started at http://127.22.108.5:38545/ using document root <none> and password file <none>
I20240503 02:11:26.359138 24449 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:26.359244 24449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:26.359563 24449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:26.362210 24449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "b6c1cc93e2b94067a17b2f284950a0a2"
format_stamp: "Formatted at 2024-05-03 02:11:26 on dist-test-slave-sc15"
I20240503 02:11:26.362996 24449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "b6c1cc93e2b94067a17b2f284950a0a2"
format_stamp: "Formatted at 2024-05-03 02:11:26 on dist-test-slave-sc15"
I20240503 02:11:26.368211 24449 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 02:11:26.371632 24463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:26.372972 24449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 02:11:26.373200 24449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "b6c1cc93e2b94067a17b2f284950a0a2"
format_stamp: "Formatted at 2024-05-03 02:11:26 on dist-test-slave-sc15"
I20240503 02:11:26.373396 24449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:26.399696 24449 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:26.400617 24449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:26.400898 24449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:26.401918 24449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:26.403465 24449 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:26.403585 24449 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:26.403707 24449 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:26.403784 24449 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:26.433696 24449 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.5:36531
I20240503 02:11:26.433753 24575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.5:36531 every 8 connection(s)
I20240503 02:11:26.435115 24449 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:11:26.441198 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 24449
I20240503 02:11:26.441412 22960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 02:11:26.448026 24576 heartbeater.cc:347] Connected to a master server at 127.22.108.62:43769
I20240503 02:11:26.448333 24576 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:26.449231 24576 heartbeater.cc:511] Master 127.22.108.62:43769 requested a full tablet report, sending...
I20240503 02:11:26.450536 23869 ts_manager.cc:194] Registered new tserver with Master: b6c1cc93e2b94067a17b2f284950a0a2 (127.22.108.5:36531)
I20240503 02:11:26.451264 23869 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.5:42779
I20240503 02:11:26.457227 22960 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 02:11:26.473055 23869 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36480:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 02:11:26.512751 23990 tablet_service.cc:1467] Processing CreateTablet for tablet 103334b7c37648dea7fe76138605e3aa (DEFAULT_TABLE table=TestTable [id=03d3848ee639486a96dbee1c7257a2e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:26.514307 23990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 103334b7c37648dea7fe76138605e3aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:26.514042 24511 tablet_service.cc:1467] Processing CreateTablet for tablet 103334b7c37648dea7fe76138605e3aa (DEFAULT_TABLE table=TestTable [id=03d3848ee639486a96dbee1c7257a2e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:26.515703 24511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 103334b7c37648dea7fe76138605e3aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:26.516093 24251 tablet_service.cc:1467] Processing CreateTablet for tablet 103334b7c37648dea7fe76138605e3aa (DEFAULT_TABLE table=TestTable [id=03d3848ee639486a96dbee1c7257a2e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:26.517521 24251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 103334b7c37648dea7fe76138605e3aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:26.525445 24594 tablet_bootstrap.cc:492] T 103334b7c37648dea7fe76138605e3aa P d60d3644181e48818a66372f8a0c9ed7: Bootstrap starting.
I20240503 02:11:26.528949 24594 tablet_bootstrap.cc:654] T 103334b7c37648dea7fe76138605e3aa P d60d3644181e48818a66372f8a0c9ed7: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:26.530252 24594 log.cc:824] T 103334b7c37648dea7fe76138605e3aa P d60d3644181e48818a66372f8a0c9ed7: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:26.531355 24595 tablet_bootstrap.cc:492] T 103334b7c37648dea7fe76138605e3aa P b6c1cc93e2b94067a17b2f284950a0a2: Bootstrap starting.
I20240503 02:11:26.536136 24595 tablet_bootstrap.cc:654] T 103334b7c37648dea7fe76138605e3aa P b6c1cc93e2b94067a17b2f284950a0a2: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:26.537849 24596 tablet_bootstrap.cc:492] T 103334b7c37648dea7fe76138605e3aa P b8aeb791035f4f04b304ef08b49dd7b6: Bootstrap starting.
I20240503 02:11:26.539359 24595 log.cc:824] T 103334b7c37648dea7fe76138605e3aa P b6c1cc93e2b94067a17b2f284950a0a2: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:26.539922 24594 tablet_bootstrap.cc:492] T 103334b7c37648dea7fe76138605e3aa P d60d3644181e48818a66372f8a0c9ed7: No bootstrap required, opened a new log
I20240503 02:11:26.540367 24594 ts_tablet_manager.cc:1367] T 103334b7c37648dea7fe76138605e3aa P d60d3644181e48818a66372f8a0c9ed7: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.011s
I20240503 02:11:26.542737 24596 tablet_bootstrap.cc:654] T 103334b7c37648dea7fe76138605e3aa P b8aeb791035f4f04b304ef08b49dd7b6: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:26.543992 24596 log.cc:824] T 103334b7c37648dea7fe76138605e3aa P b8aeb791035f4f04b304ef08b49dd7b6: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:26.549232 24595 tablet_bootstrap.cc:492] T 103334b7c37648dea7fe76138605e3aa P b6c1cc93e2b94067a17b2f284950a0a2: No bootstrap required, opened a new log
I20240503 02:11:26.549854 24595 ts_tablet_manager.cc:1367] T 103334b7c37648dea7fe76138605e3aa P b6c1cc93e2b94067a17b2f284950a0a2: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.003s
I20240503 02:11:26.550968 24594 raft_consensus.cc:348] T 103334b7c37648dea7fe76138605e3aa P d60d3644181e48818a66372f8a0c9ed7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8aeb791035f4f04b304ef08b49dd7b6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 33185 } } peers { permanent_uuid: "d60d3644181e48818a66372f8a0c9ed7" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36335 } } peers { permanent_uuid: "b6c1cc93e2b94067a17b2f284950a0a2" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 36531 } }
I20240503 02:11:26.551437 24594 raft_consensus.cc:374] T 103334b7c37648dea7fe76138605e3aa P d60d3644181e48818a66372f8a0c9ed7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:26.551577 24594 raft_consensus.cc:729] T 103334b7c37648dea7fe76138605e3aa P d60d3644181e48818a66372f8a0c9ed7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d60d3644181e48818a66372f8a0c9ed7, State: Initialized, Role: FOLLOWER
I20240503 02:11:26.552325 24594 consensus_queue.cc:260] T 103334b7c37648dea7fe76138605e3aa P d60d3644181e48818a66372f8a0c9ed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8aeb791035f4f04b304ef08b49dd7b6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 33185 } } peers { permanent_uuid: "d60d3644181e48818a66372f8a0c9ed7" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36335 } } peers { permanent_uuid: "b6c1cc93e2b94067a17b2f284950a0a2" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 36531 } }
I20240503 02:11:26.554591 24594 ts_tablet_manager.cc:1398] T 103334b7c37648dea7fe76138605e3aa P d60d3644181e48818a66372f8a0c9ed7: Time spent starting tablet: real 0.014s user 0.011s sys 0.001s
I20240503 02:11:26.555048 24316 heartbeater.cc:503] Master 127.22.108.62:43769 was elected leader, sending a full tablet report...
I20240503 02:11:26.555843 24596 tablet_bootstrap.cc:492] T 103334b7c37648dea7fe76138605e3aa P b8aeb791035f4f04b304ef08b49dd7b6: No bootstrap required, opened a new log
I20240503 02:11:26.556190 24596 ts_tablet_manager.cc:1367] T 103334b7c37648dea7fe76138605e3aa P b8aeb791035f4f04b304ef08b49dd7b6: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.010s
I20240503 02:11:26.559060 24595 raft_consensus.cc:348] T 103334b7c37648dea7fe76138605e3aa P b6c1cc93e2b94067a17b2f284950a0a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8aeb791035f4f04b304ef08b49dd7b6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 33185 } } peers { permanent_uuid: "d60d3644181e48818a66372f8a0c9ed7" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36335 } } peers { permanent_uuid: "b6c1cc93e2b94067a17b2f284950a0a2" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 36531 } }
I20240503 02:11:26.559500 24595 raft_consensus.cc:374] T 103334b7c37648dea7fe76138605e3aa P b6c1cc93e2b94067a17b2f284950a0a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:26.559615 24595 raft_consensus.cc:729] T 103334b7c37648dea7fe76138605e3aa P b6c1cc93e2b94067a17b2f284950a0a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6c1cc93e2b94067a17b2f284950a0a2, State: Initialized, Role: FOLLOWER
I20240503 02:11:26.560282 24595 consensus_queue.cc:260] T 103334b7c37648dea7fe76138605e3aa P b6c1cc93e2b94067a17b2f284950a0a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8aeb791035f4f04b304ef08b49dd7b6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 33185 } } peers { permanent_uuid: "d60d3644181e48818a66372f8a0c9ed7" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36335 } } peers { permanent_uuid: "b6c1cc93e2b94067a17b2f284950a0a2" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 36531 } }
I20240503 02:11:26.561400 24576 heartbeater.cc:503] Master 127.22.108.62:43769 was elected leader, sending a full tablet report...
I20240503 02:11:26.562088 24595 ts_tablet_manager.cc:1398] T 103334b7c37648dea7fe76138605e3aa P b6c1cc93e2b94067a17b2f284950a0a2: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20240503 02:11:26.564499 24596 raft_consensus.cc:348] T 103334b7c37648dea7fe76138605e3aa P b8aeb791035f4f04b304ef08b49dd7b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8aeb791035f4f04b304ef08b49dd7b6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 33185 } } peers { permanent_uuid: "d60d3644181e48818a66372f8a0c9ed7" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36335 } } peers { permanent_uuid: "b6c1cc93e2b94067a17b2f284950a0a2" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 36531 } }
I20240503 02:11:26.564847 24596 raft_consensus.cc:374] T 103334b7c37648dea7fe76138605e3aa P b8aeb791035f4f04b304ef08b49dd7b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:26.564958 24596 raft_consensus.cc:729] T 103334b7c37648dea7fe76138605e3aa P b8aeb791035f4f04b304ef08b49dd7b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8aeb791035f4f04b304ef08b49dd7b6, State: Initialized, Role: FOLLOWER
I20240503 02:11:26.565717 24596 consensus_queue.cc:260] T 103334b7c37648dea7fe76138605e3aa P b8aeb791035f4f04b304ef08b49dd7b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8aeb791035f4f04b304ef08b49dd7b6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 33185 } } peers { permanent_uuid: "d60d3644181e48818a66372f8a0c9ed7" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36335 } } peers { permanent_uuid: "b6c1cc93e2b94067a17b2f284950a0a2" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 36531 } }
I20240503 02:11:26.567134 24596 ts_tablet_manager.cc:1398] T 103334b7c37648dea7fe76138605e3aa P b8aeb791035f4f04b304ef08b49dd7b6: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
W20240503 02:11:26.625680 24056 tablet.cc:2369] T 103334b7c37648dea7fe76138605e3aa P d60d3644181e48818a66372f8a0c9ed7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:11:26.686894 24577 tablet.cc:2369] T 103334b7c37648dea7fe76138605e3aa P b6c1cc93e2b94067a17b2f284950a0a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:11:26.788101 24317 tablet.cc:2369] T 103334b7c37648dea7fe76138605e3aa P b8aeb791035f4f04b304ef08b49dd7b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:11:26.846396 24600 raft_consensus.cc:482] T 103334b7c37648dea7fe76138605e3aa P d60d3644181e48818a66372f8a0c9ed7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:26.846662 24600 raft_consensus.cc:504] T 103334b7c37648dea7fe76138605e3aa P d60d3644181e48818a66372f8a0c9ed7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8aeb791035f4f04b304ef08b49dd7b6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 33185 } } peers { permanent_uuid: "d60d3644181e48818a66372f8a0c9ed7" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36335 } } peers { permanent_uuid: "b6c1cc93e2b94067a17b2f284950a0a2" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 36531 } }
I20240503 02:11:26.848333 24600 leader_election.cc:288] T 103334b7c37648dea7fe76138605e3aa P d60d3644181e48818a66372f8a0c9ed7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8aeb791035f4f04b304ef08b49dd7b6 (127.22.108.3:33185), b6c1cc93e2b94067a17b2f284950a0a2 (127.22.108.5:36531)
I20240503 02:11:26.855227 24601 raft_consensus.cc:482] T 103334b7c37648dea7fe76138605e3aa P b6c1cc93e2b94067a17b2f284950a0a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:26.855504 24601 raft_consensus.cc:504] T 103334b7c37648dea7fe76138605e3aa P b6c1cc93e2b94067a17b2f284950a0a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8aeb791035f4f04b304ef08b49dd7b6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 33185 } } peers { permanent_uuid: "d60d3644181e48818a66372f8a0c9ed7" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36335 } } peers { permanent_uuid: "b6c1cc93e2b94067a17b2f284950a0a2" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 36531 } }
I20240503 02:11:26.856161 24271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "103334b7c37648dea7fe76138605e3aa" candidate_uuid: "d60d3644181e48818a66372f8a0c9ed7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8aeb791035f4f04b304ef08b49dd7b6" is_pre_election: true
I20240503 02:11:26.856124 24531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "103334b7c37648dea7fe76138605e3aa" candidate_uuid: "d60d3644181e48818a66372f8a0c9ed7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6c1cc93e2b94067a17b2f284950a0a2" is_pre_election: true
I20240503 02:11:26.856541 24531 raft_consensus.cc:2454] T 103334b7c37648dea7fe76138605e3aa P b6c1cc93e2b94067a17b2f284950a0a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d60d3644181e48818a66372f8a0c9ed7 in term 0.
I20240503 02:11:26.856619 24271 raft_consensus.cc:2454] T 103334b7c37648dea7fe76138605e3aa P b8aeb791035f4f04b304ef08b49dd7b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d60d3644181e48818a66372f8a0c9ed7 in term 0.
I20240503 02:11:26.857213 23943 leader_election.cc:302] T 103334b7c37648dea7fe76138605e3aa P d60d3644181e48818a66372f8a0c9ed7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6c1cc93e2b94067a17b2f284950a0a2, d60d3644181e48818a66372f8a0c9ed7; no voters:
I20240503 02:11:26.857524 24601 leader_election.cc:288] T 103334b7c37648dea7fe76138605e3aa P b6c1cc93e2b94067a17b2f284950a0a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8aeb791035f4f04b304ef08b49dd7b6 (127.22.108.3:33185), d60d3644181e48818a66372f8a0c9ed7 (127.22.108.1:36335)
I20240503 02:11:26.858368 24600 raft_consensus.cc:2781] T 103334b7c37648dea7fe76138605e3aa P d60d3644181e48818a66372f8a0c9ed7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:26.858505 24600 raft_consensus.cc:482] T 103334b7c37648dea7fe76138605e3aa P d60d3644181e48818a66372f8a0c9ed7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:26.858606 24600 raft_consensus.cc:3037] T 103334b7c37648dea7fe76138605e3aa P d60d3644181e48818a66372f8a0c9ed7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:26.861240 24600 raft_consensus.cc:504] T 103334b7c37648dea7fe76138605e3aa P d60d3644181e48818a66372f8a0c9ed7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8aeb791035f4f04b304ef08b49dd7b6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 33185 } } peers { permanent_uuid: "d60d3644181e48818a66372f8a0c9ed7" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36335 } } peers { permanent_uuid: "b6c1cc93e2b94067a17b2f284950a0a2" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 36531 } }
I20240503 02:11:26.862849 24271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "103334b7c37648dea7fe76138605e3aa" candidate_uuid: "d60d3644181e48818a66372f8a0c9ed7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8aeb791035f4f04b304ef08b49dd7b6"
I20240503 02:11:26.862866 24531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "103334b7c37648dea7fe76138605e3aa" candidate_uuid: "d60d3644181e48818a66372f8a0c9ed7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6c1cc93e2b94067a17b2f284950a0a2"
I20240503 02:11:26.863051 24531 raft_consensus.cc:3037] T 103334b7c37648dea7fe76138605e3aa P b6c1cc93e2b94067a17b2f284950a0a2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:26.863067 24271 raft_consensus.cc:3037] T 103334b7c37648dea7fe76138605e3aa P b8aeb791035f4f04b304ef08b49dd7b6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:26.865432 24531 raft_consensus.cc:2454] T 103334b7c37648dea7fe76138605e3aa P b6c1cc93e2b94067a17b2f284950a0a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d60d3644181e48818a66372f8a0c9ed7 in term 1.
I20240503 02:11:26.865815 23943 leader_election.cc:302] T 103334b7c37648dea7fe76138605e3aa P d60d3644181e48818a66372f8a0c9ed7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6c1cc93e2b94067a17b2f284950a0a2, d60d3644181e48818a66372f8a0c9ed7; no voters:
I20240503 02:11:26.865929 24271 raft_consensus.cc:2454] T 103334b7c37648dea7fe76138605e3aa P b8aeb791035f4f04b304ef08b49dd7b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d60d3644181e48818a66372f8a0c9ed7 in term 1.
I20240503 02:11:26.866560 24270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "103334b7c37648dea7fe76138605e3aa" candidate_uuid: "b6c1cc93e2b94067a17b2f284950a0a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8aeb791035f4f04b304ef08b49dd7b6" is_pre_election: true
I20240503 02:11:26.866763 24606 raft_consensus.cc:2781] T 103334b7c37648dea7fe76138605e3aa P d60d3644181e48818a66372f8a0c9ed7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:26.866592 24010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "103334b7c37648dea7fe76138605e3aa" candidate_uuid: "b6c1cc93e2b94067a17b2f284950a0a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d60d3644181e48818a66372f8a0c9ed7" is_pre_election: true
I20240503 02:11:26.866874 24270 raft_consensus.cc:2379] T 103334b7c37648dea7fe76138605e3aa P b8aeb791035f4f04b304ef08b49dd7b6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b6c1cc93e2b94067a17b2f284950a0a2 in current term 1: Already voted for candidate d60d3644181e48818a66372f8a0c9ed7 in this term.
I20240503 02:11:26.867341 24606 raft_consensus.cc:686] T 103334b7c37648dea7fe76138605e3aa P d60d3644181e48818a66372f8a0c9ed7 [term 1 LEADER]: Becoming Leader. State: Replica: d60d3644181e48818a66372f8a0c9ed7, State: Running, Role: LEADER
I20240503 02:11:26.865849 24600 leader_election.cc:288] T 103334b7c37648dea7fe76138605e3aa P d60d3644181e48818a66372f8a0c9ed7 [CANDIDATE]: Term 1 election: Requested vote from peers b8aeb791035f4f04b304ef08b49dd7b6 (127.22.108.3:33185), b6c1cc93e2b94067a17b2f284950a0a2 (127.22.108.5:36531)
I20240503 02:11:26.868029 24606 consensus_queue.cc:237] T 103334b7c37648dea7fe76138605e3aa P d60d3644181e48818a66372f8a0c9ed7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8aeb791035f4f04b304ef08b49dd7b6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 33185 } } peers { permanent_uuid: "d60d3644181e48818a66372f8a0c9ed7" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36335 } } peers { permanent_uuid: "b6c1cc93e2b94067a17b2f284950a0a2" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 36531 } }
I20240503 02:11:26.871899 24464 leader_election.cc:302] T 103334b7c37648dea7fe76138605e3aa P b6c1cc93e2b94067a17b2f284950a0a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b6c1cc93e2b94067a17b2f284950a0a2; no voters: b8aeb791035f4f04b304ef08b49dd7b6, d60d3644181e48818a66372f8a0c9ed7
I20240503 02:11:26.872275 24601 raft_consensus.cc:2726] T 103334b7c37648dea7fe76138605e3aa P b6c1cc93e2b94067a17b2f284950a0a2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:11:26.872967 23868 catalog_manager.cc:5484] T 103334b7c37648dea7fe76138605e3aa P d60d3644181e48818a66372f8a0c9ed7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d60d3644181e48818a66372f8a0c9ed7 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "d60d3644181e48818a66372f8a0c9ed7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8aeb791035f4f04b304ef08b49dd7b6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 33185 } } peers { permanent_uuid: "d60d3644181e48818a66372f8a0c9ed7" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36335 } } peers { permanent_uuid: "b6c1cc93e2b94067a17b2f284950a0a2" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 36531 } } }
I20240503 02:11:26.962406 22960 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 02:11:26.964473 22960 ts_itest-base.cc:233] Waiting for 1 tablets on tserver d60d3644181e48818a66372f8a0c9ed7 to finish bootstrapping
I20240503 02:11:26.972918 22960 ts_itest-base.cc:233] Waiting for 0 tablets on tserver cacd0decd933496798fe28a7a793f947 to finish bootstrapping
I20240503 02:11:26.982086 22960 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b8aeb791035f4f04b304ef08b49dd7b6 to finish bootstrapping
I20240503 02:11:26.991667 22960 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 4a36fb8d35744a5fb5505391524a2cf0 to finish bootstrapping
I20240503 02:11:27.000123 22960 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b6c1cc93e2b94067a17b2f284950a0a2 to finish bootstrapping
I20240503 02:11:27.000312 24446 heartbeater.cc:503] Master 127.22.108.62:43769 was elected leader, sending a full tablet report...
I20240503 02:11:27.007544 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 23928
I20240503 02:11:27.008479 24050 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:27.097409 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 23928
I20240503 02:11:27.582140 24251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:27.588255 24121 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:27.590970 24511 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:27.598814 24381 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:27.724529 24511 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:27.726220 24381 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:27.730310 24251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:27.730406 24121 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:27.895325 24381 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:27.898202 24121 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:27.899425 24511 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:27.909525 24251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:28.481972 24381 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:28.493602 24511 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:28.500268 24251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:28.501506 24121 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:28.626786 24381 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:28.639361 24121 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:28.640900 24251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:28.643998 24511 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:28.773732 24789 raft_consensus.cc:482] T 103334b7c37648dea7fe76138605e3aa P b8aeb791035f4f04b304ef08b49dd7b6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:28.773927 24789 raft_consensus.cc:504] T 103334b7c37648dea7fe76138605e3aa P b8aeb791035f4f04b304ef08b49dd7b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8aeb791035f4f04b304ef08b49dd7b6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 33185 } } peers { permanent_uuid: "d60d3644181e48818a66372f8a0c9ed7" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36335 } } peers { permanent_uuid: "b6c1cc93e2b94067a17b2f284950a0a2" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 36531 } }
I20240503 02:11:28.775705 24789 leader_election.cc:288] T 103334b7c37648dea7fe76138605e3aa P b8aeb791035f4f04b304ef08b49dd7b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d60d3644181e48818a66372f8a0c9ed7 (127.22.108.1:36335), b6c1cc93e2b94067a17b2f284950a0a2 (127.22.108.5:36531)
W20240503 02:11:28.795060 24204 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.1:36335: connect: Connection refused (error 111)
I20240503 02:11:28.805667 24531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "103334b7c37648dea7fe76138605e3aa" candidate_uuid: "b8aeb791035f4f04b304ef08b49dd7b6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6c1cc93e2b94067a17b2f284950a0a2" is_pre_election: true
I20240503 02:11:28.805991 24531 raft_consensus.cc:2454] T 103334b7c37648dea7fe76138605e3aa P b6c1cc93e2b94067a17b2f284950a0a2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8aeb791035f4f04b304ef08b49dd7b6 in term 1.
W20240503 02:11:28.806205 24204 leader_election.cc:334] T 103334b7c37648dea7fe76138605e3aa P b8aeb791035f4f04b304ef08b49dd7b6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d60d3644181e48818a66372f8a0c9ed7 (127.22.108.1:36335): Network error: Client connection negotiation failed: client connection to 127.22.108.1:36335: connect: Connection refused (error 111)
I20240503 02:11:28.807587 24381 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:28.808406 24511 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:28.811403 24204 leader_election.cc:302] T 103334b7c37648dea7fe76138605e3aa P b8aeb791035f4f04b304ef08b49dd7b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b6c1cc93e2b94067a17b2f284950a0a2, b8aeb791035f4f04b304ef08b49dd7b6; no voters: d60d3644181e48818a66372f8a0c9ed7
I20240503 02:11:28.811832 24789 raft_consensus.cc:2781] T 103334b7c37648dea7fe76138605e3aa P b8aeb791035f4f04b304ef08b49dd7b6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:11:28.811952 24789 raft_consensus.cc:482] T 103334b7c37648dea7fe76138605e3aa P b8aeb791035f4f04b304ef08b49dd7b6 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:28.812021 24789 raft_consensus.cc:3037] T 103334b7c37648dea7fe76138605e3aa P b8aeb791035f4f04b304ef08b49dd7b6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:28.814951 24789 raft_consensus.cc:504] T 103334b7c37648dea7fe76138605e3aa P b8aeb791035f4f04b304ef08b49dd7b6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8aeb791035f4f04b304ef08b49dd7b6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 33185 } } peers { permanent_uuid: "d60d3644181e48818a66372f8a0c9ed7" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36335 } } peers { permanent_uuid: "b6c1cc93e2b94067a17b2f284950a0a2" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 36531 } }
I20240503 02:11:28.815661 24789 leader_election.cc:288] T 103334b7c37648dea7fe76138605e3aa P b8aeb791035f4f04b304ef08b49dd7b6 [CANDIDATE]: Term 2 election: Requested vote from peers d60d3644181e48818a66372f8a0c9ed7 (127.22.108.1:36335), b6c1cc93e2b94067a17b2f284950a0a2 (127.22.108.5:36531)
I20240503 02:11:28.819396 24531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "103334b7c37648dea7fe76138605e3aa" candidate_uuid: "b8aeb791035f4f04b304ef08b49dd7b6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6c1cc93e2b94067a17b2f284950a0a2"
I20240503 02:11:28.819749 24531 raft_consensus.cc:3037] T 103334b7c37648dea7fe76138605e3aa P b6c1cc93e2b94067a17b2f284950a0a2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:28.819998 24121 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:28.821807 24251 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:28.823062 24531 raft_consensus.cc:2454] T 103334b7c37648dea7fe76138605e3aa P b6c1cc93e2b94067a17b2f284950a0a2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8aeb791035f4f04b304ef08b49dd7b6 in term 2.
I20240503 02:11:28.823577 24204 leader_election.cc:302] T 103334b7c37648dea7fe76138605e3aa P b8aeb791035f4f04b304ef08b49dd7b6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6c1cc93e2b94067a17b2f284950a0a2, b8aeb791035f4f04b304ef08b49dd7b6; no voters:
I20240503 02:11:28.823885 24789 raft_consensus.cc:2781] T 103334b7c37648dea7fe76138605e3aa P b8aeb791035f4f04b304ef08b49dd7b6 [term 2 FOLLOWER]: Leader election won for term 2
W20240503 02:11:28.833452 24204 leader_election.cc:334] T 103334b7c37648dea7fe76138605e3aa P b8aeb791035f4f04b304ef08b49dd7b6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d60d3644181e48818a66372f8a0c9ed7 (127.22.108.1:36335): Network error: Client connection negotiation failed: client connection to 127.22.108.1:36335: connect: Connection refused (error 111)
I20240503 02:11:28.835880 24789 raft_consensus.cc:686] T 103334b7c37648dea7fe76138605e3aa P b8aeb791035f4f04b304ef08b49dd7b6 [term 2 LEADER]: Becoming Leader. State: Replica: b8aeb791035f4f04b304ef08b49dd7b6, State: Running, Role: LEADER
I20240503 02:11:28.836498 24789 consensus_queue.cc:237] T 103334b7c37648dea7fe76138605e3aa P b8aeb791035f4f04b304ef08b49dd7b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8aeb791035f4f04b304ef08b49dd7b6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 33185 } } peers { permanent_uuid: "d60d3644181e48818a66372f8a0c9ed7" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36335 } } peers { permanent_uuid: "b6c1cc93e2b94067a17b2f284950a0a2" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 36531 } }
I20240503 02:11:28.843752 23869 catalog_manager.cc:5484] T 103334b7c37648dea7fe76138605e3aa P b8aeb791035f4f04b304ef08b49dd7b6 reported cstate change: term changed from 1 to 2, leader changed from d60d3644181e48818a66372f8a0c9ed7 (127.22.108.1) to b8aeb791035f4f04b304ef08b49dd7b6 (127.22.108.3). New cstate: current_term: 2 leader_uuid: "b8aeb791035f4f04b304ef08b49dd7b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8aeb791035f4f04b304ef08b49dd7b6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 33185 } } peers { permanent_uuid: "d60d3644181e48818a66372f8a0c9ed7" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36335 } } peers { permanent_uuid: "b6c1cc93e2b94067a17b2f284950a0a2" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 36531 } } }
I20240503 02:11:28.981796 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 24058
I20240503 02:11:28.985504 24181 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:29.082576 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 24058
I20240503 02:11:29.100677 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 24189
I20240503 02:11:29.101758 24311 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:29.200418 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 24189
I20240503 02:11:29.219331 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 24319
I20240503 02:11:29.220367 24441 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:29.310650 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 24319
I20240503 02:11:29.328090 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 24449
I20240503 02:11:29.329111 24570 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:29.425299 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 24449
I20240503 02:11:29.443284 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 23838
I20240503 02:11:29.444422 23899 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:29.557924 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 23838
2024-05-03T02:11:29Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.TabletServerIsDown/0 (5914 ms)
[----------] 1 test from RebalanceStartCriteriaTest (5914 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/0
I20240503 02:11:29.600541 22960 test_util.cc:268] Using random seed: 1021181956
I20240503 02:11:29.602396 22960 ts_itest-base.cc:112] Starting cluster with:
I20240503 02:11:29.602459 22960 ts_itest-base.cc:113] --------------
I20240503 02:11:29.602504 22960 ts_itest-base.cc:114] 7 tablet servers
I20240503 02:11:29.602543 22960 ts_itest-base.cc:115] 1 replicas per TS
I20240503 02:11:29.602599 22960 ts_itest-base.cc:116] --------------
2024-05-03T02:11:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:11:29Z Disabled control of system clock
I20240503 02:11:29.625870 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.108.62:38409
--webserver_interface=127.22.108.62
--webserver_port=0
--builtin_ntp_servers=127.22.108.20:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.108.62:38409
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240503 02:11:29.944602 24818 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 02:11:29.944936 24818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:29.945037 24818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:29.945158 24818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:29.952584 24818 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:11:29.952690 24818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:11:29.952775 24818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:29.952818 24818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:11:29.952905 24818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:11:29.961877 24818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.22.108.62:38409
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.108.62:38409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.108.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:29.963224 24818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:29.964967 24818 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:29.975289 24823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:29.975875 24824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:29.976632 24826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:29.977659 24818 server_base.cc:1034] running on GCE node
I20240503 02:11:29.978328 24818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:29.979593 24818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:29.980806 24818 hybrid_clock.cc:648] HybridClock initialized: now 1714702289980750 us; error 32 us; skew 500 ppm
I20240503 02:11:29.981137 24818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:29.988852 24818 webserver.cc:460] Webserver started at http://127.22.108.62:35719/ using document root <none> and password file <none>
I20240503 02:11:29.989593 24818 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:29.989722 24818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:29.990026 24818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:29.992647 24818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "5a79f2a86741450aa9992d60c94daf6f"
format_stamp: "Formatted at 2024-05-03 02:11:29 on dist-test-slave-sc15"
I20240503 02:11:29.993467 24818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "5a79f2a86741450aa9992d60c94daf6f"
format_stamp: "Formatted at 2024-05-03 02:11:29 on dist-test-slave-sc15"
I20240503 02:11:29.998410 24818 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240503 02:11:30.001762 24832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:30.003042 24818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:11:30.003196 24818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/wal
uuid: "5a79f2a86741450aa9992d60c94daf6f"
format_stamp: "Formatted at 2024-05-03 02:11:29 on dist-test-slave-sc15"
I20240503 02:11:30.003358 24818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:30.049633 24818 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:30.050477 24818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:30.050769 24818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:30.068140 24818 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.62:38409
I20240503 02:11:30.068171 24883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.62:38409 every 8 connection(s)
I20240503 02:11:30.069499 24818 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 02:11:30.073004 24884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:30.074028 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 24818
I20240503 02:11:30.074209 22960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 02:11:30.083173 24884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a79f2a86741450aa9992d60c94daf6f: Bootstrap starting.
I20240503 02:11:30.086813 24884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a79f2a86741450aa9992d60c94daf6f: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:30.087939 24884 log.cc:824] T 00000000000000000000000000000000 P 5a79f2a86741450aa9992d60c94daf6f: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:30.090471 24884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a79f2a86741450aa9992d60c94daf6f: No bootstrap required, opened a new log
I20240503 02:11:30.096439 24884 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a79f2a86741450aa9992d60c94daf6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a79f2a86741450aa9992d60c94daf6f" member_type: VOTER last_known_addr { host: "127.22.108.62" port: 38409 } }
I20240503 02:11:30.096776 24884 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a79f2a86741450aa9992d60c94daf6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:30.096870 24884 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a79f2a86741450aa9992d60c94daf6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a79f2a86741450aa9992d60c94daf6f, State: Initialized, Role: FOLLOWER
I20240503 02:11:30.097508 24884 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a79f2a86741450aa9992d60c94daf6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a79f2a86741450aa9992d60c94daf6f" member_type: VOTER last_known_addr { host: "127.22.108.62" port: 38409 } }
I20240503 02:11:30.097698 24884 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a79f2a86741450aa9992d60c94daf6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:30.097816 24884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a79f2a86741450aa9992d60c94daf6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:30.097946 24884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a79f2a86741450aa9992d60c94daf6f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:30.100123 24884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a79f2a86741450aa9992d60c94daf6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a79f2a86741450aa9992d60c94daf6f" member_type: VOTER last_known_addr { host: "127.22.108.62" port: 38409 } }
I20240503 02:11:30.100567 24884 leader_election.cc:302] T 00000000000000000000000000000000 P 5a79f2a86741450aa9992d60c94daf6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a79f2a86741450aa9992d60c94daf6f; no voters:
I20240503 02:11:30.101176 24884 leader_election.cc:288] T 00000000000000000000000000000000 P 5a79f2a86741450aa9992d60c94daf6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:30.101276 24889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a79f2a86741450aa9992d60c94daf6f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:30.101900 24889 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a79f2a86741450aa9992d60c94daf6f [term 1 LEADER]: Becoming Leader. State: Replica: 5a79f2a86741450aa9992d60c94daf6f, State: Running, Role: LEADER
I20240503 02:11:30.102455 24889 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a79f2a86741450aa9992d60c94daf6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a79f2a86741450aa9992d60c94daf6f" member_type: VOTER last_known_addr { host: "127.22.108.62" port: 38409 } }
I20240503 02:11:30.103031 24884 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a79f2a86741450aa9992d60c94daf6f [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:11:30.105145 24891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a79f2a86741450aa9992d60c94daf6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a79f2a86741450aa9992d60c94daf6f. Latest consensus state: current_term: 1 leader_uuid: "5a79f2a86741450aa9992d60c94daf6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a79f2a86741450aa9992d60c94daf6f" member_type: VOTER last_known_addr { host: "127.22.108.62" port: 38409 } } }
I20240503 02:11:30.105175 24890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a79f2a86741450aa9992d60c94daf6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a79f2a86741450aa9992d60c94daf6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a79f2a86741450aa9992d60c94daf6f" member_type: VOTER last_known_addr { host: "127.22.108.62" port: 38409 } } }
I20240503 02:11:30.105494 24891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a79f2a86741450aa9992d60c94daf6f [sys.catalog]: This master's current role is: LEADER
I20240503 02:11:30.105547 24890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a79f2a86741450aa9992d60c94daf6f [sys.catalog]: This master's current role is: LEADER
I20240503 02:11:30.110361 24895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:11:30.114537 24895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:11:30.122180 24895 catalog_manager.cc:1347] Generated new cluster ID: 2aa674d7907d496eb22fc955ed8326b2
I20240503 02:11:30.122294 24895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:11:30.147650 24895 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:11:30.148820 24895 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:11:30.156486 24895 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 5a79f2a86741450aa9992d60c94daf6f: Generated new TSK 0
I20240503 02:11:30.157328 24895 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:11:30.164618 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.1:0
--local_ip_for_outbound_sockets=127.22.108.1
--webserver_interface=127.22.108.1
--webserver_port=0
--tserver_master_addrs=127.22.108.62:38409
--builtin_ntp_servers=127.22.108.20:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 02:11:30.506225 24908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:30.506536 24908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:30.506623 24908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:30.514338 24908 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:11:30.514463 24908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:30.514623 24908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.1
I20240503 02:11:30.523921 24908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.108.1
--webserver_port=0
--tserver_master_addrs=127.22.108.62:38409
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.1
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:30.525398 24908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:30.527246 24908 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:30.539261 24913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:30.539832 24914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:30.540772 24916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:30.541155 24908 server_base.cc:1034] running on GCE node
I20240503 02:11:30.541720 24908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:30.542625 24908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:30.543812 24908 hybrid_clock.cc:648] HybridClock initialized: now 1714702290543772 us; error 36 us; skew 500 ppm
I20240503 02:11:30.544101 24908 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:30.546296 24908 webserver.cc:460] Webserver started at http://127.22.108.1:39741/ using document root <none> and password file <none>
I20240503 02:11:30.546970 24908 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:30.547080 24908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:30.547376 24908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:30.549786 24908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "308436a6f7c7456b99971c666d3bd049"
format_stamp: "Formatted at 2024-05-03 02:11:30 on dist-test-slave-sc15"
I20240503 02:11:30.550523 24908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "308436a6f7c7456b99971c666d3bd049"
format_stamp: "Formatted at 2024-05-03 02:11:30 on dist-test-slave-sc15"
I20240503 02:11:30.555296 24908 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20240503 02:11:30.558343 24922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:30.559514 24908 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 02:11:30.559659 24908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "308436a6f7c7456b99971c666d3bd049"
format_stamp: "Formatted at 2024-05-03 02:11:30 on dist-test-slave-sc15"
I20240503 02:11:30.559834 24908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:30.579970 24908 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:30.580677 24908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:30.580941 24908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:30.581837 24908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:30.583341 24908 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:30.583451 24908 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:30.583573 24908 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:30.583662 24908 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:30.613499 24908 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.1:40941
I20240503 02:11:30.613548 25034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.1:40941 every 8 connection(s)
I20240503 02:11:30.614722 24908 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 02:11:30.622521 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 24908
I20240503 02:11:30.622742 22960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 02:11:30.626449 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.2:0
--local_ip_for_outbound_sockets=127.22.108.2
--webserver_interface=127.22.108.2
--webserver_port=0
--tserver_master_addrs=127.22.108.62:38409
--builtin_ntp_servers=127.22.108.20:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:11:30.627512 25035 heartbeater.cc:347] Connected to a master server at 127.22.108.62:38409
I20240503 02:11:30.627780 25035 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:30.628460 25035 heartbeater.cc:511] Master 127.22.108.62:38409 requested a full tablet report, sending...
I20240503 02:11:30.630472 24849 ts_manager.cc:194] Registered new tserver with Master: 308436a6f7c7456b99971c666d3bd049 (127.22.108.1:40941)
I20240503 02:11:30.632266 24849 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.1:36049
W20240503 02:11:30.955147 25039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:30.955468 25039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:30.955556 25039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:30.963160 25039 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:11:30.963335 25039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:30.963523 25039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.2
I20240503 02:11:30.973086 25039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.108.2
--webserver_port=0
--tserver_master_addrs=127.22.108.62:38409
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.2
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:30.974511 25039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:30.976276 25039 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:30.988281 25044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:30.989197 25045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:30.989698 25047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:30.990391 25039 server_base.cc:1034] running on GCE node
I20240503 02:11:30.991050 25039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:30.992045 25039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:30.993263 25039 hybrid_clock.cc:648] HybridClock initialized: now 1714702290993217 us; error 43 us; skew 500 ppm
I20240503 02:11:30.993582 25039 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:30.995803 25039 webserver.cc:460] Webserver started at http://127.22.108.2:37467/ using document root <none> and password file <none>
I20240503 02:11:30.996536 25039 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:30.996651 25039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:30.996977 25039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:30.999636 25039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "22b0b1ae6c504301ad5170c6bdcd179e"
format_stamp: "Formatted at 2024-05-03 02:11:30 on dist-test-slave-sc15"
I20240503 02:11:31.000526 25039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "22b0b1ae6c504301ad5170c6bdcd179e"
format_stamp: "Formatted at 2024-05-03 02:11:30 on dist-test-slave-sc15"
I20240503 02:11:31.005831 25039 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240503 02:11:31.009481 25053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:31.010897 25039 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240503 02:11:31.011049 25039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "22b0b1ae6c504301ad5170c6bdcd179e"
format_stamp: "Formatted at 2024-05-03 02:11:30 on dist-test-slave-sc15"
I20240503 02:11:31.011224 25039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:31.027024 25039 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:31.027841 25039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:31.028086 25039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:31.028981 25039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:31.030486 25039 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:31.030568 25039 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:31.030659 25039 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:31.030712 25039 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:31.059476 25039 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.2:36213
I20240503 02:11:31.059528 25165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.2:36213 every 8 connection(s)
I20240503 02:11:31.060864 25039 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 02:11:31.066730 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 25039
I20240503 02:11:31.066890 22960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 02:11:31.070149 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.3:0
--local_ip_for_outbound_sockets=127.22.108.3
--webserver_interface=127.22.108.3
--webserver_port=0
--tserver_master_addrs=127.22.108.62:38409
--builtin_ntp_servers=127.22.108.20:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:11:31.075314 25166 heartbeater.cc:347] Connected to a master server at 127.22.108.62:38409
I20240503 02:11:31.075666 25166 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:31.076557 25166 heartbeater.cc:511] Master 127.22.108.62:38409 requested a full tablet report, sending...
I20240503 02:11:31.078193 24849 ts_manager.cc:194] Registered new tserver with Master: 22b0b1ae6c504301ad5170c6bdcd179e (127.22.108.2:36213)
I20240503 02:11:31.079421 24849 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.2:33191
W20240503 02:11:31.404726 25169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:31.405092 25169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:31.405213 25169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:31.413095 25169 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:11:31.413221 25169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:31.413341 25169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.3
I20240503 02:11:31.422677 25169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.108.3
--webserver_port=0
--tserver_master_addrs=127.22.108.62:38409
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.3
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:31.424091 25169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:31.425923 25169 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:31.437754 25174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:31.438764 25175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:31.439395 25177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:31.439981 25169 server_base.cc:1034] running on GCE node
I20240503 02:11:31.440567 25169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:31.441556 25169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:31.442761 25169 hybrid_clock.cc:648] HybridClock initialized: now 1714702291442711 us; error 39 us; skew 500 ppm
I20240503 02:11:31.443061 25169 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:31.445294 25169 webserver.cc:460] Webserver started at http://127.22.108.3:36617/ using document root <none> and password file <none>
I20240503 02:11:31.446023 25169 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:31.446125 25169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:31.446430 25169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:31.448957 25169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "b95ea99cb72f49c6b42c7e8e293cfc27"
format_stamp: "Formatted at 2024-05-03 02:11:31 on dist-test-slave-sc15"
I20240503 02:11:31.449745 25169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "b95ea99cb72f49c6b42c7e8e293cfc27"
format_stamp: "Formatted at 2024-05-03 02:11:31 on dist-test-slave-sc15"
I20240503 02:11:31.454617 25169 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240503 02:11:31.457808 25183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:31.459249 25169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:11:31.459424 25169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b95ea99cb72f49c6b42c7e8e293cfc27"
format_stamp: "Formatted at 2024-05-03 02:11:31 on dist-test-slave-sc15"
I20240503 02:11:31.459621 25169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:31.477687 25169 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:31.478605 25169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:31.478895 25169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:31.479902 25169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:31.481562 25169 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:31.481666 25169 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:31.481786 25169 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:31.481868 25169 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:31.514608 25169 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.3:36823
I20240503 02:11:31.514668 25295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.3:36823 every 8 connection(s)
I20240503 02:11:31.516063 25169 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:11:31.521521 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 25169
I20240503 02:11:31.521739 22960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 02:11:31.525502 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.4:0
--local_ip_for_outbound_sockets=127.22.108.4
--webserver_interface=127.22.108.4
--webserver_port=0
--tserver_master_addrs=127.22.108.62:38409
--builtin_ntp_servers=127.22.108.20:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:11:31.531368 25296 heartbeater.cc:347] Connected to a master server at 127.22.108.62:38409
I20240503 02:11:31.531694 25296 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:31.532495 25296 heartbeater.cc:511] Master 127.22.108.62:38409 requested a full tablet report, sending...
I20240503 02:11:31.534027 24849 ts_manager.cc:194] Registered new tserver with Master: b95ea99cb72f49c6b42c7e8e293cfc27 (127.22.108.3:36823)
I20240503 02:11:31.534847 24849 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.3:43731
I20240503 02:11:31.635128 25035 heartbeater.cc:503] Master 127.22.108.62:38409 was elected leader, sending a full tablet report...
W20240503 02:11:31.863318 25299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:31.863646 25299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:31.863802 25299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:31.871515 25299 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:11:31.871668 25299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:31.871862 25299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.4
I20240503 02:11:31.883005 25299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.108.4
--webserver_port=0
--tserver_master_addrs=127.22.108.62:38409
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.4
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:31.884353 25299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:31.886255 25299 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:31.898205 25304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:31.898769 25305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:31.899637 25299 server_base.cc:1034] running on GCE node
W20240503 02:11:31.900347 25307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:31.901100 25299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:31.902093 25299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:31.903326 25299 hybrid_clock.cc:648] HybridClock initialized: now 1714702291903262 us; error 49 us; skew 500 ppm
I20240503 02:11:31.903751 25299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:31.906056 25299 webserver.cc:460] Webserver started at http://127.22.108.4:45785/ using document root <none> and password file <none>
I20240503 02:11:31.906790 25299 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:31.906898 25299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:31.907281 25299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:31.910022 25299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "0163beb78bfd4d399a70ebacb8f9aa9c"
format_stamp: "Formatted at 2024-05-03 02:11:31 on dist-test-slave-sc15"
I20240503 02:11:31.910888 25299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "0163beb78bfd4d399a70ebacb8f9aa9c"
format_stamp: "Formatted at 2024-05-03 02:11:31 on dist-test-slave-sc15"
I20240503 02:11:31.916131 25299 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 02:11:31.919497 25313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:31.920727 25299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 02:11:31.920897 25299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "0163beb78bfd4d399a70ebacb8f9aa9c"
format_stamp: "Formatted at 2024-05-03 02:11:31 on dist-test-slave-sc15"
I20240503 02:11:31.921109 25299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:31.932976 25299 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:31.933779 25299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:31.934047 25299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:31.935000 25299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:31.936419 25299 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:31.936523 25299 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:31.936643 25299 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:31.936723 25299 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:31.968432 25299 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.4:45401
I20240503 02:11:31.968487 25425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.4:45401 every 8 connection(s)
I20240503 02:11:31.969894 25299 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:11:31.974645 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 25299
I20240503 02:11:31.974838 22960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 02:11:31.978492 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.5:0
--local_ip_for_outbound_sockets=127.22.108.5
--webserver_interface=127.22.108.5
--webserver_port=0
--tserver_master_addrs=127.22.108.62:38409
--builtin_ntp_servers=127.22.108.20:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:11:31.984440 25426 heartbeater.cc:347] Connected to a master server at 127.22.108.62:38409
I20240503 02:11:31.984736 25426 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:31.985489 25426 heartbeater.cc:511] Master 127.22.108.62:38409 requested a full tablet report, sending...
I20240503 02:11:31.986821 24849 ts_manager.cc:194] Registered new tserver with Master: 0163beb78bfd4d399a70ebacb8f9aa9c (127.22.108.4:45401)
I20240503 02:11:31.987561 24849 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.4:41625
I20240503 02:11:32.082546 25166 heartbeater.cc:503] Master 127.22.108.62:38409 was elected leader, sending a full tablet report...
W20240503 02:11:32.312985 25429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:32.313366 25429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:32.313513 25429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:32.320969 25429 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:11:32.321154 25429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:32.321316 25429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.5
I20240503 02:11:32.330621 25429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.108.5
--webserver_port=0
--tserver_master_addrs=127.22.108.62:38409
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.5
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:32.331979 25429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:32.333807 25429 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:32.345497 25434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:32.346223 25437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:32.347249 25435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:32.348213 25429 server_base.cc:1034] running on GCE node
I20240503 02:11:32.348834 25429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:32.349824 25429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:32.351024 25429 hybrid_clock.cc:648] HybridClock initialized: now 1714702292350978 us; error 42 us; skew 500 ppm
I20240503 02:11:32.351331 25429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:32.353616 25429 webserver.cc:460] Webserver started at http://127.22.108.5:41907/ using document root <none> and password file <none>
I20240503 02:11:32.354344 25429 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:32.354470 25429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:32.354769 25429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:32.357329 25429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "1b2a71b48d4e40679504230af08279e9"
format_stamp: "Formatted at 2024-05-03 02:11:32 on dist-test-slave-sc15"
I20240503 02:11:32.358057 25429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "1b2a71b48d4e40679504230af08279e9"
format_stamp: "Formatted at 2024-05-03 02:11:32 on dist-test-slave-sc15"
I20240503 02:11:32.362813 25429 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 02:11:32.366359 25443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:32.367563 25429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:11:32.367704 25429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "1b2a71b48d4e40679504230af08279e9"
format_stamp: "Formatted at 2024-05-03 02:11:32 on dist-test-slave-sc15"
I20240503 02:11:32.367882 25429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:32.385393 25429 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:32.386232 25429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:32.386499 25429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:32.387425 25429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:32.388741 25429 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:32.388820 25429 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:32.388897 25429 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:32.388948 25429 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:32.420111 25429 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.5:40741
I20240503 02:11:32.420157 25555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.5:40741 every 8 connection(s)
I20240503 02:11:32.421473 25429 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:11:32.427716 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 25429
I20240503 02:11:32.427922 22960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 02:11:32.431299 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.6:0
--local_ip_for_outbound_sockets=127.22.108.6
--webserver_interface=127.22.108.6
--webserver_port=0
--tserver_master_addrs=127.22.108.62:38409
--builtin_ntp_servers=127.22.108.20:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:11:32.435060 25556 heartbeater.cc:347] Connected to a master server at 127.22.108.62:38409
I20240503 02:11:32.435393 25556 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:32.436161 25556 heartbeater.cc:511] Master 127.22.108.62:38409 requested a full tablet report, sending...
I20240503 02:11:32.437497 24849 ts_manager.cc:194] Registered new tserver with Master: 1b2a71b48d4e40679504230af08279e9 (127.22.108.5:40741)
I20240503 02:11:32.438220 24849 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.5:44617
I20240503 02:11:32.537854 25296 heartbeater.cc:503] Master 127.22.108.62:38409 was elected leader, sending a full tablet report...
W20240503 02:11:32.757771 25559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:32.758100 25559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:32.758239 25559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:32.765978 25559 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:11:32.766130 25559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:32.766299 25559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.6
I20240503 02:11:32.775503 25559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.22.108.6
--webserver_port=0
--tserver_master_addrs=127.22.108.62:38409
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.6
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:32.776856 25559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:32.778582 25559 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:32.791133 25565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:32.791869 25567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:32.792083 25564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:32.793146 25559 server_base.cc:1034] running on GCE node
I20240503 02:11:32.793709 25559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:32.794632 25559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:32.795832 25559 hybrid_clock.cc:648] HybridClock initialized: now 1714702292795787 us; error 35 us; skew 500 ppm
I20240503 02:11:32.796144 25559 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:32.798338 25559 webserver.cc:460] Webserver started at http://127.22.108.6:35125/ using document root <none> and password file <none>
I20240503 02:11:32.799077 25559 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:32.799191 25559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:32.799504 25559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:32.802181 25559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "6c704dfc19814db996a1c0e363e760ef"
format_stamp: "Formatted at 2024-05-03 02:11:32 on dist-test-slave-sc15"
I20240503 02:11:32.802984 25559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "6c704dfc19814db996a1c0e363e760ef"
format_stamp: "Formatted at 2024-05-03 02:11:32 on dist-test-slave-sc15"
I20240503 02:11:32.808359 25559 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20240503 02:11:32.811830 25573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:32.813196 25559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:11:32.813330 25559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6c704dfc19814db996a1c0e363e760ef"
format_stamp: "Formatted at 2024-05-03 02:11:32 on dist-test-slave-sc15"
I20240503 02:11:32.813503 25559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:32.834044 25559 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:32.834890 25559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:32.835172 25559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:32.836120 25559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:32.837623 25559 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:32.837706 25559 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:32.837816 25559 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:32.837888 25559 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:32.868816 25559 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.6:43731
I20240503 02:11:32.868867 25685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.6:43731 every 8 connection(s)
I20240503 02:11:32.870203 25559 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 02:11:32.880404 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 25559
I20240503 02:11:32.880579 22960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 02:11:32.882172 25686 heartbeater.cc:347] Connected to a master server at 127.22.108.62:38409
I20240503 02:11:32.882452 25686 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:32.883323 25686 heartbeater.cc:511] Master 127.22.108.62:38409 requested a full tablet report, sending...
I20240503 02:11:32.884006 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.7:0
--local_ip_for_outbound_sockets=127.22.108.7
--webserver_interface=127.22.108.7
--webserver_port=0
--tserver_master_addrs=127.22.108.62:38409
--builtin_ntp_servers=127.22.108.20:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:11:32.884778 24849 ts_manager.cc:194] Registered new tserver with Master: 6c704dfc19814db996a1c0e363e760ef (127.22.108.6:43731)
I20240503 02:11:32.885541 24849 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.6:56503
I20240503 02:11:32.989998 25426 heartbeater.cc:503] Master 127.22.108.62:38409 was elected leader, sending a full tablet report...
W20240503 02:11:33.219659 25689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:33.219981 25689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:33.220131 25689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:33.228281 25689 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:11:33.228464 25689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:33.228633 25689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.7
I20240503 02:11:33.237776 25689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.22.108.7
--webserver_port=0
--tserver_master_addrs=127.22.108.62:38409
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.7
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:33.239151 25689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:33.240949 25689 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:33.254154 25694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:33.254289 25695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:33.254992 25697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:33.256098 25689 server_base.cc:1034] running on GCE node
I20240503 02:11:33.256695 25689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:33.257670 25689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:33.258875 25689 hybrid_clock.cc:648] HybridClock initialized: now 1714702293258835 us; error 33 us; skew 500 ppm
I20240503 02:11:33.259191 25689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:33.261950 25689 webserver.cc:460] Webserver started at http://127.22.108.7:45859/ using document root <none> and password file <none>
I20240503 02:11:33.262660 25689 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:33.262771 25689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:33.263073 25689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:33.265805 25689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "03afb03fe005462eb07ada37fe23c3cf"
format_stamp: "Formatted at 2024-05-03 02:11:33 on dist-test-slave-sc15"
I20240503 02:11:33.266558 25689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "03afb03fe005462eb07ada37fe23c3cf"
format_stamp: "Formatted at 2024-05-03 02:11:33 on dist-test-slave-sc15"
I20240503 02:11:33.271839 25689 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 02:11:33.275379 25703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:33.276736 25689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240503 02:11:33.276903 25689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "03afb03fe005462eb07ada37fe23c3cf"
format_stamp: "Formatted at 2024-05-03 02:11:33 on dist-test-slave-sc15"
I20240503 02:11:33.277150 25689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:33.310379 25689 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:33.311273 25689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:33.311568 25689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:33.312530 25689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:33.314054 25689 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:33.314142 25689 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:33.314258 25689 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:33.314327 25689 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:33.349704 25689 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.7:34851
I20240503 02:11:33.349761 25815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.7:34851 every 8 connection(s)
I20240503 02:11:33.351004 25689 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 02:11:33.353344 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 25689
I20240503 02:11:33.353538 22960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 02:11:33.364224 25816 heartbeater.cc:347] Connected to a master server at 127.22.108.62:38409
I20240503 02:11:33.364531 25816 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:33.365351 25816 heartbeater.cc:511] Master 127.22.108.62:38409 requested a full tablet report, sending...
I20240503 02:11:33.366631 24849 ts_manager.cc:194] Registered new tserver with Master: 03afb03fe005462eb07ada37fe23c3cf (127.22.108.7:34851)
I20240503 02:11:33.367431 24849 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.7:47721
I20240503 02:11:33.369561 22960 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 02:11:33.385550 24849 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44436:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 02:11:33.413110 25101 tablet_service.cc:1467] Processing CreateTablet for tablet ee1a68079bfa435ab820967fb894d3f1 (DEFAULT_TABLE table=TestTable [id=152bc29db2bb47d98fa21091fc2aaf3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:33.414526 25101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee1a68079bfa435ab820967fb894d3f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:33.425462 25830 tablet_bootstrap.cc:492] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e: Bootstrap starting.
I20240503 02:11:33.429087 25830 tablet_bootstrap.cc:654] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:33.430052 25830 log.cc:824] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:33.432320 25830 tablet_bootstrap.cc:492] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e: No bootstrap required, opened a new log
I20240503 02:11:33.432540 25830 ts_tablet_manager.cc:1367] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240503 02:11:33.438603 25830 raft_consensus.cc:348] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } }
I20240503 02:11:33.438901 25830 raft_consensus.cc:374] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:33.438966 25830 raft_consensus.cc:729] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b0b1ae6c504301ad5170c6bdcd179e, State: Initialized, Role: FOLLOWER
I20240503 02:11:33.439538 25830 consensus_queue.cc:260] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } }
I20240503 02:11:33.439719 25830 raft_consensus.cc:388] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:33.439810 25830 raft_consensus.cc:482] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:33.439930 25830 raft_consensus.cc:3037] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:33.441010 25556 heartbeater.cc:503] Master 127.22.108.62:38409 was elected leader, sending a full tablet report...
I20240503 02:11:33.442796 25830 raft_consensus.cc:504] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } }
I20240503 02:11:33.443253 25830 leader_election.cc:302] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22b0b1ae6c504301ad5170c6bdcd179e; no voters:
I20240503 02:11:33.443838 25830 leader_election.cc:288] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:33.443966 25832 raft_consensus.cc:2781] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:33.444550 25832 raft_consensus.cc:686] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 LEADER]: Becoming Leader. State: Replica: 22b0b1ae6c504301ad5170c6bdcd179e, State: Running, Role: LEADER
I20240503 02:11:33.445178 25832 consensus_queue.cc:237] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } }
I20240503 02:11:33.445564 25830 ts_tablet_manager.cc:1398] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20240503 02:11:33.450119 24848 catalog_manager.cc:5484] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e reported cstate change: term changed from 0 to 1, leader changed from <none> to 22b0b1ae6c504301ad5170c6bdcd179e (127.22.108.2). New cstate: current_term: 1 leader_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } }
I20240503 02:11:33.458460 22960 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 02:11:33.460289 22960 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 308436a6f7c7456b99971c666d3bd049 to finish bootstrapping
I20240503 02:11:33.469781 22960 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 22b0b1ae6c504301ad5170c6bdcd179e to finish bootstrapping
I20240503 02:11:33.476934 22960 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b95ea99cb72f49c6b42c7e8e293cfc27 to finish bootstrapping
I20240503 02:11:33.485666 22960 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 0163beb78bfd4d399a70ebacb8f9aa9c to finish bootstrapping
I20240503 02:11:33.493530 22960 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 1b2a71b48d4e40679504230af08279e9 to finish bootstrapping
I20240503 02:11:33.502077 22960 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 6c704dfc19814db996a1c0e363e760ef to finish bootstrapping
I20240503 02:11:33.511420 22960 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 03afb03fe005462eb07ada37fe23c3cf to finish bootstrapping
I20240503 02:11:33.521620 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 25169
I20240503 02:11:33.522570 25291 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:33.602097 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 25169
W20240503 02:11:33.619959 24836 connection.cc:535] server connection from 127.22.108.3:43731 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:11:33.620298 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 25299
I20240503 02:11:33.621510 25421 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:33.700857 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 25299
I20240503 02:11:33.716629 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 25429
I20240503 02:11:33.717758 25551 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:33.796347 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 25429
I20240503 02:11:33.812228 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 25559
I20240503 02:11:33.813297 25681 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:33.891948 25686 heartbeater.cc:503] Master 127.22.108.62:38409 was elected leader, sending a full tablet report...
I20240503 02:11:33.892961 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 25559
W20240503 02:11:33.908484 24834 connection.cc:535] server connection from 127.22.108.6:56503 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:11:33.908799 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 25689
I20240503 02:11:33.909806 25811 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:33.988806 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 25689
I20240503 02:11:37.756372 24848 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44436:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:11:37.776258 24970 tablet_service.cc:1467] Processing CreateTablet for tablet a141985c93da484494d5f82c478d0bc3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=adca86e19a68413aa3c2e0c525d683a5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:37.776227 24968 tablet_service.cc:1467] Processing CreateTablet for tablet 742a25555073462faaabd4af58f42017 (DEFAULT_TABLE table=rebalance_test_table_0 [id=adca86e19a68413aa3c2e0c525d683a5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:37.776316 24969 tablet_service.cc:1467] Processing CreateTablet for tablet e1093d22a1dc4e0f81fd3df39bf03265 (DEFAULT_TABLE table=rebalance_test_table_0 [id=adca86e19a68413aa3c2e0c525d683a5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:37.778126 24968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 742a25555073462faaabd4af58f42017. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:37.778666 24970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a141985c93da484494d5f82c478d0bc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:37.778928 24969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1093d22a1dc4e0f81fd3df39bf03265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:37.791004 25855 tablet_bootstrap.cc:492] T 742a25555073462faaabd4af58f42017 P 308436a6f7c7456b99971c666d3bd049: Bootstrap starting.
I20240503 02:11:37.794796 25855 tablet_bootstrap.cc:654] T 742a25555073462faaabd4af58f42017 P 308436a6f7c7456b99971c666d3bd049: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:37.795936 25855 log.cc:824] T 742a25555073462faaabd4af58f42017 P 308436a6f7c7456b99971c666d3bd049: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:37.798763 25855 tablet_bootstrap.cc:492] T 742a25555073462faaabd4af58f42017 P 308436a6f7c7456b99971c666d3bd049: No bootstrap required, opened a new log
I20240503 02:11:37.799049 25855 ts_tablet_manager.cc:1367] T 742a25555073462faaabd4af58f42017 P 308436a6f7c7456b99971c666d3bd049: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20240503 02:11:37.805452 25855 raft_consensus.cc:348] T 742a25555073462faaabd4af58f42017 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:37.805836 25855 raft_consensus.cc:374] T 742a25555073462faaabd4af58f42017 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:37.805944 25855 raft_consensus.cc:729] T 742a25555073462faaabd4af58f42017 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 308436a6f7c7456b99971c666d3bd049, State: Initialized, Role: FOLLOWER
I20240503 02:11:37.806607 25855 consensus_queue.cc:260] T 742a25555073462faaabd4af58f42017 P 308436a6f7c7456b99971c666d3bd049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:37.806875 25855 raft_consensus.cc:388] T 742a25555073462faaabd4af58f42017 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:37.807029 25855 raft_consensus.cc:482] T 742a25555073462faaabd4af58f42017 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:37.807181 25855 raft_consensus.cc:3037] T 742a25555073462faaabd4af58f42017 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:37.812196 25855 raft_consensus.cc:504] T 742a25555073462faaabd4af58f42017 P 308436a6f7c7456b99971c666d3bd049 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:37.812819 25855 leader_election.cc:302] T 742a25555073462faaabd4af58f42017 P 308436a6f7c7456b99971c666d3bd049 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 308436a6f7c7456b99971c666d3bd049; no voters:
I20240503 02:11:37.813565 25855 leader_election.cc:288] T 742a25555073462faaabd4af58f42017 P 308436a6f7c7456b99971c666d3bd049 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:37.813690 25857 raft_consensus.cc:2781] T 742a25555073462faaabd4af58f42017 P 308436a6f7c7456b99971c666d3bd049 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:37.814251 25857 raft_consensus.cc:686] T 742a25555073462faaabd4af58f42017 P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Becoming Leader. State: Replica: 308436a6f7c7456b99971c666d3bd049, State: Running, Role: LEADER
I20240503 02:11:37.815382 25857 consensus_queue.cc:237] T 742a25555073462faaabd4af58f42017 P 308436a6f7c7456b99971c666d3bd049 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:37.816372 25855 ts_tablet_manager.cc:1398] T 742a25555073462faaabd4af58f42017 P 308436a6f7c7456b99971c666d3bd049: Time spent starting tablet: real 0.017s user 0.010s sys 0.005s
I20240503 02:11:37.816855 25855 tablet_bootstrap.cc:492] T e1093d22a1dc4e0f81fd3df39bf03265 P 308436a6f7c7456b99971c666d3bd049: Bootstrap starting.
I20240503 02:11:37.818568 25855 tablet_bootstrap.cc:654] T e1093d22a1dc4e0f81fd3df39bf03265 P 308436a6f7c7456b99971c666d3bd049: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:37.820817 25855 tablet_bootstrap.cc:492] T e1093d22a1dc4e0f81fd3df39bf03265 P 308436a6f7c7456b99971c666d3bd049: No bootstrap required, opened a new log
I20240503 02:11:37.820958 25855 ts_tablet_manager.cc:1367] T e1093d22a1dc4e0f81fd3df39bf03265 P 308436a6f7c7456b99971c666d3bd049: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:11:37.820807 24848 catalog_manager.cc:5484] T 742a25555073462faaabd4af58f42017 P 308436a6f7c7456b99971c666d3bd049 reported cstate change: term changed from 0 to 1, leader changed from <none> to 308436a6f7c7456b99971c666d3bd049 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "308436a6f7c7456b99971c666d3bd049" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } }
I20240503 02:11:37.821955 25855 raft_consensus.cc:348] T e1093d22a1dc4e0f81fd3df39bf03265 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:37.822083 25855 raft_consensus.cc:374] T e1093d22a1dc4e0f81fd3df39bf03265 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:37.822151 25855 raft_consensus.cc:729] T e1093d22a1dc4e0f81fd3df39bf03265 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 308436a6f7c7456b99971c666d3bd049, State: Initialized, Role: FOLLOWER
I20240503 02:11:37.822363 25855 consensus_queue.cc:260] T e1093d22a1dc4e0f81fd3df39bf03265 P 308436a6f7c7456b99971c666d3bd049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:37.822517 25855 raft_consensus.cc:388] T e1093d22a1dc4e0f81fd3df39bf03265 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:37.822593 25855 raft_consensus.cc:482] T e1093d22a1dc4e0f81fd3df39bf03265 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:37.822698 25855 raft_consensus.cc:3037] T e1093d22a1dc4e0f81fd3df39bf03265 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:37.916314 25855 raft_consensus.cc:504] T e1093d22a1dc4e0f81fd3df39bf03265 P 308436a6f7c7456b99971c666d3bd049 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:37.916718 25855 leader_election.cc:302] T e1093d22a1dc4e0f81fd3df39bf03265 P 308436a6f7c7456b99971c666d3bd049 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 308436a6f7c7456b99971c666d3bd049; no voters:
I20240503 02:11:37.916952 25855 leader_election.cc:288] T e1093d22a1dc4e0f81fd3df39bf03265 P 308436a6f7c7456b99971c666d3bd049 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:37.917114 25857 raft_consensus.cc:2781] T e1093d22a1dc4e0f81fd3df39bf03265 P 308436a6f7c7456b99971c666d3bd049 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:37.917433 25857 raft_consensus.cc:686] T e1093d22a1dc4e0f81fd3df39bf03265 P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Becoming Leader. State: Replica: 308436a6f7c7456b99971c666d3bd049, State: Running, Role: LEADER
I20240503 02:11:37.917532 25855 ts_tablet_manager.cc:1398] T e1093d22a1dc4e0f81fd3df39bf03265 P 308436a6f7c7456b99971c666d3bd049: Time spent starting tablet: real 0.096s user 0.005s sys 0.000s
I20240503 02:11:37.917747 25857 consensus_queue.cc:237] T e1093d22a1dc4e0f81fd3df39bf03265 P 308436a6f7c7456b99971c666d3bd049 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:37.917905 25855 tablet_bootstrap.cc:492] T a141985c93da484494d5f82c478d0bc3 P 308436a6f7c7456b99971c666d3bd049: Bootstrap starting.
I20240503 02:11:37.920114 25855 tablet_bootstrap.cc:654] T a141985c93da484494d5f82c478d0bc3 P 308436a6f7c7456b99971c666d3bd049: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:37.922464 24848 catalog_manager.cc:5484] T e1093d22a1dc4e0f81fd3df39bf03265 P 308436a6f7c7456b99971c666d3bd049 reported cstate change: term changed from 0 to 1, leader changed from <none> to 308436a6f7c7456b99971c666d3bd049 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "308436a6f7c7456b99971c666d3bd049" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } }
I20240503 02:11:37.923112 25855 tablet_bootstrap.cc:492] T a141985c93da484494d5f82c478d0bc3 P 308436a6f7c7456b99971c666d3bd049: No bootstrap required, opened a new log
I20240503 02:11:37.923277 25855 ts_tablet_manager.cc:1367] T a141985c93da484494d5f82c478d0bc3 P 308436a6f7c7456b99971c666d3bd049: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 02:11:37.923877 25855 raft_consensus.cc:348] T a141985c93da484494d5f82c478d0bc3 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:37.924052 25855 raft_consensus.cc:374] T a141985c93da484494d5f82c478d0bc3 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:37.924139 25855 raft_consensus.cc:729] T a141985c93da484494d5f82c478d0bc3 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 308436a6f7c7456b99971c666d3bd049, State: Initialized, Role: FOLLOWER
I20240503 02:11:37.924394 25855 consensus_queue.cc:260] T a141985c93da484494d5f82c478d0bc3 P 308436a6f7c7456b99971c666d3bd049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:37.924571 25855 raft_consensus.cc:388] T a141985c93da484494d5f82c478d0bc3 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:37.924681 25855 raft_consensus.cc:482] T a141985c93da484494d5f82c478d0bc3 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:37.924777 25855 raft_consensus.cc:3037] T a141985c93da484494d5f82c478d0bc3 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:38.020763 25855 raft_consensus.cc:504] T a141985c93da484494d5f82c478d0bc3 P 308436a6f7c7456b99971c666d3bd049 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:38.021189 25855 leader_election.cc:302] T a141985c93da484494d5f82c478d0bc3 P 308436a6f7c7456b99971c666d3bd049 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 308436a6f7c7456b99971c666d3bd049; no voters:
I20240503 02:11:38.021485 25855 leader_election.cc:288] T a141985c93da484494d5f82c478d0bc3 P 308436a6f7c7456b99971c666d3bd049 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:38.021620 25857 raft_consensus.cc:2781] T a141985c93da484494d5f82c478d0bc3 P 308436a6f7c7456b99971c666d3bd049 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:38.022055 25855 ts_tablet_manager.cc:1398] T a141985c93da484494d5f82c478d0bc3 P 308436a6f7c7456b99971c666d3bd049: Time spent starting tablet: real 0.099s user 0.005s sys 0.000s
I20240503 02:11:38.022145 25857 raft_consensus.cc:686] T a141985c93da484494d5f82c478d0bc3 P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Becoming Leader. State: Replica: 308436a6f7c7456b99971c666d3bd049, State: Running, Role: LEADER
I20240503 02:11:38.022414 25857 consensus_queue.cc:237] T a141985c93da484494d5f82c478d0bc3 P 308436a6f7c7456b99971c666d3bd049 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:38.024731 24848 catalog_manager.cc:5484] T a141985c93da484494d5f82c478d0bc3 P 308436a6f7c7456b99971c666d3bd049 reported cstate change: term changed from 0 to 1, leader changed from <none> to 308436a6f7c7456b99971c666d3bd049 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "308436a6f7c7456b99971c666d3bd049" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } }
W20240503 02:11:38.416191 25866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:38.416632 25866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:38.424225 25866 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 121.831 ms
time per row: 0.0609153 ms
I20240503 02:11:38.708333 24847 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44436:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:11:38.717106 25101 tablet_service.cc:1467] Processing CreateTablet for tablet bb1a9efdfd96421ea17c41cfa8977f36 (DEFAULT_TABLE table=rebalance_test_table_1 [id=dcb3b303097f4c479078780f22a99899]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:38.717618 25101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb1a9efdfd96421ea17c41cfa8977f36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:38.717640 25100 tablet_service.cc:1467] Processing CreateTablet for tablet 92f95fea174e43c9bc3aa019d2191e51 (DEFAULT_TABLE table=rebalance_test_table_1 [id=dcb3b303097f4c479078780f22a99899]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:38.717866 25099 tablet_service.cc:1467] Processing CreateTablet for tablet 4ce4d66aff0e48deb5513001f1d81963 (DEFAULT_TABLE table=rebalance_test_table_1 [id=dcb3b303097f4c479078780f22a99899]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:38.718104 25100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92f95fea174e43c9bc3aa019d2191e51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:38.718443 25099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ce4d66aff0e48deb5513001f1d81963. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:38.724373 25907 tablet_bootstrap.cc:492] T 92f95fea174e43c9bc3aa019d2191e51 P 22b0b1ae6c504301ad5170c6bdcd179e: Bootstrap starting.
I20240503 02:11:38.726073 25907 tablet_bootstrap.cc:654] T 92f95fea174e43c9bc3aa019d2191e51 P 22b0b1ae6c504301ad5170c6bdcd179e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:38.727910 25907 tablet_bootstrap.cc:492] T 92f95fea174e43c9bc3aa019d2191e51 P 22b0b1ae6c504301ad5170c6bdcd179e: No bootstrap required, opened a new log
I20240503 02:11:38.728032 25907 ts_tablet_manager.cc:1367] T 92f95fea174e43c9bc3aa019d2191e51 P 22b0b1ae6c504301ad5170c6bdcd179e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:11:38.728812 25907 raft_consensus.cc:348] T 92f95fea174e43c9bc3aa019d2191e51 P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } }
I20240503 02:11:38.728951 25907 raft_consensus.cc:374] T 92f95fea174e43c9bc3aa019d2191e51 P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:38.729064 25907 raft_consensus.cc:729] T 92f95fea174e43c9bc3aa019d2191e51 P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b0b1ae6c504301ad5170c6bdcd179e, State: Initialized, Role: FOLLOWER
I20240503 02:11:38.729295 25907 consensus_queue.cc:260] T 92f95fea174e43c9bc3aa019d2191e51 P 22b0b1ae6c504301ad5170c6bdcd179e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } }
I20240503 02:11:38.729460 25907 raft_consensus.cc:388] T 92f95fea174e43c9bc3aa019d2191e51 P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:38.729542 25907 raft_consensus.cc:482] T 92f95fea174e43c9bc3aa019d2191e51 P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:38.729671 25907 raft_consensus.cc:3037] T 92f95fea174e43c9bc3aa019d2191e51 P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:38.749548 25907 raft_consensus.cc:504] T 92f95fea174e43c9bc3aa019d2191e51 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } }
I20240503 02:11:38.749927 25907 leader_election.cc:302] T 92f95fea174e43c9bc3aa019d2191e51 P 22b0b1ae6c504301ad5170c6bdcd179e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22b0b1ae6c504301ad5170c6bdcd179e; no voters:
I20240503 02:11:38.750419 25907 leader_election.cc:288] T 92f95fea174e43c9bc3aa019d2191e51 P 22b0b1ae6c504301ad5170c6bdcd179e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:38.750607 25909 raft_consensus.cc:2781] T 92f95fea174e43c9bc3aa019d2191e51 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:38.751387 25907 ts_tablet_manager.cc:1398] T 92f95fea174e43c9bc3aa019d2191e51 P 22b0b1ae6c504301ad5170c6bdcd179e: Time spent starting tablet: real 0.023s user 0.002s sys 0.005s
I20240503 02:11:38.751761 25907 tablet_bootstrap.cc:492] T bb1a9efdfd96421ea17c41cfa8977f36 P 22b0b1ae6c504301ad5170c6bdcd179e: Bootstrap starting.
I20240503 02:11:38.751730 25909 raft_consensus.cc:686] T 92f95fea174e43c9bc3aa019d2191e51 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 LEADER]: Becoming Leader. State: Replica: 22b0b1ae6c504301ad5170c6bdcd179e, State: Running, Role: LEADER
I20240503 02:11:38.752087 25909 consensus_queue.cc:237] T 92f95fea174e43c9bc3aa019d2191e51 P 22b0b1ae6c504301ad5170c6bdcd179e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } }
I20240503 02:11:38.753407 25907 tablet_bootstrap.cc:654] T bb1a9efdfd96421ea17c41cfa8977f36 P 22b0b1ae6c504301ad5170c6bdcd179e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:38.755903 25907 tablet_bootstrap.cc:492] T bb1a9efdfd96421ea17c41cfa8977f36 P 22b0b1ae6c504301ad5170c6bdcd179e: No bootstrap required, opened a new log
I20240503 02:11:38.756129 25907 ts_tablet_manager.cc:1367] T bb1a9efdfd96421ea17c41cfa8977f36 P 22b0b1ae6c504301ad5170c6bdcd179e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:11:38.756315 24847 catalog_manager.cc:5484] T 92f95fea174e43c9bc3aa019d2191e51 P 22b0b1ae6c504301ad5170c6bdcd179e reported cstate change: term changed from 0 to 1, leader changed from <none> to 22b0b1ae6c504301ad5170c6bdcd179e (127.22.108.2). New cstate: current_term: 1 leader_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } }
I20240503 02:11:38.756816 25907 raft_consensus.cc:348] T bb1a9efdfd96421ea17c41cfa8977f36 P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } }
I20240503 02:11:38.756975 25907 raft_consensus.cc:374] T bb1a9efdfd96421ea17c41cfa8977f36 P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:38.757057 25907 raft_consensus.cc:729] T bb1a9efdfd96421ea17c41cfa8977f36 P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b0b1ae6c504301ad5170c6bdcd179e, State: Initialized, Role: FOLLOWER
I20240503 02:11:38.757249 25907 consensus_queue.cc:260] T bb1a9efdfd96421ea17c41cfa8977f36 P 22b0b1ae6c504301ad5170c6bdcd179e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } }
I20240503 02:11:38.757387 25907 raft_consensus.cc:388] T bb1a9efdfd96421ea17c41cfa8977f36 P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:38.757458 25907 raft_consensus.cc:482] T bb1a9efdfd96421ea17c41cfa8977f36 P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:38.757526 25907 raft_consensus.cc:3037] T bb1a9efdfd96421ea17c41cfa8977f36 P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:38.851913 25907 raft_consensus.cc:504] T bb1a9efdfd96421ea17c41cfa8977f36 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } }
I20240503 02:11:38.852283 25907 leader_election.cc:302] T bb1a9efdfd96421ea17c41cfa8977f36 P 22b0b1ae6c504301ad5170c6bdcd179e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22b0b1ae6c504301ad5170c6bdcd179e; no voters:
I20240503 02:11:38.852550 25907 leader_election.cc:288] T bb1a9efdfd96421ea17c41cfa8977f36 P 22b0b1ae6c504301ad5170c6bdcd179e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:38.852663 25909 raft_consensus.cc:2781] T bb1a9efdfd96421ea17c41cfa8977f36 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:38.852927 25909 raft_consensus.cc:686] T bb1a9efdfd96421ea17c41cfa8977f36 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 LEADER]: Becoming Leader. State: Replica: 22b0b1ae6c504301ad5170c6bdcd179e, State: Running, Role: LEADER
I20240503 02:11:38.853266 25907 ts_tablet_manager.cc:1398] T bb1a9efdfd96421ea17c41cfa8977f36 P 22b0b1ae6c504301ad5170c6bdcd179e: Time spent starting tablet: real 0.097s user 0.005s sys 0.000s
I20240503 02:11:38.853269 25909 consensus_queue.cc:237] T bb1a9efdfd96421ea17c41cfa8977f36 P 22b0b1ae6c504301ad5170c6bdcd179e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } }
I20240503 02:11:38.853726 25907 tablet_bootstrap.cc:492] T 4ce4d66aff0e48deb5513001f1d81963 P 22b0b1ae6c504301ad5170c6bdcd179e: Bootstrap starting.
I20240503 02:11:38.856021 25907 tablet_bootstrap.cc:654] T 4ce4d66aff0e48deb5513001f1d81963 P 22b0b1ae6c504301ad5170c6bdcd179e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:38.857949 24847 catalog_manager.cc:5484] T bb1a9efdfd96421ea17c41cfa8977f36 P 22b0b1ae6c504301ad5170c6bdcd179e reported cstate change: term changed from 0 to 1, leader changed from <none> to 22b0b1ae6c504301ad5170c6bdcd179e (127.22.108.2). New cstate: current_term: 1 leader_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } }
I20240503 02:11:38.858246 25907 tablet_bootstrap.cc:492] T 4ce4d66aff0e48deb5513001f1d81963 P 22b0b1ae6c504301ad5170c6bdcd179e: No bootstrap required, opened a new log
I20240503 02:11:38.858383 25907 ts_tablet_manager.cc:1367] T 4ce4d66aff0e48deb5513001f1d81963 P 22b0b1ae6c504301ad5170c6bdcd179e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20240503 02:11:38.859272 25907 raft_consensus.cc:348] T 4ce4d66aff0e48deb5513001f1d81963 P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } }
I20240503 02:11:38.859426 25907 raft_consensus.cc:374] T 4ce4d66aff0e48deb5513001f1d81963 P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:38.859506 25907 raft_consensus.cc:729] T 4ce4d66aff0e48deb5513001f1d81963 P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b0b1ae6c504301ad5170c6bdcd179e, State: Initialized, Role: FOLLOWER
I20240503 02:11:38.859752 25907 consensus_queue.cc:260] T 4ce4d66aff0e48deb5513001f1d81963 P 22b0b1ae6c504301ad5170c6bdcd179e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } }
I20240503 02:11:38.859959 25907 raft_consensus.cc:388] T 4ce4d66aff0e48deb5513001f1d81963 P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:38.860045 25907 raft_consensus.cc:482] T 4ce4d66aff0e48deb5513001f1d81963 P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:38.860148 25907 raft_consensus.cc:3037] T 4ce4d66aff0e48deb5513001f1d81963 P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:38.954864 25907 raft_consensus.cc:504] T 4ce4d66aff0e48deb5513001f1d81963 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } }
I20240503 02:11:38.955268 25907 leader_election.cc:302] T 4ce4d66aff0e48deb5513001f1d81963 P 22b0b1ae6c504301ad5170c6bdcd179e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22b0b1ae6c504301ad5170c6bdcd179e; no voters:
I20240503 02:11:38.955508 25907 leader_election.cc:288] T 4ce4d66aff0e48deb5513001f1d81963 P 22b0b1ae6c504301ad5170c6bdcd179e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:38.955695 25909 raft_consensus.cc:2781] T 4ce4d66aff0e48deb5513001f1d81963 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:38.956110 25907 ts_tablet_manager.cc:1398] T 4ce4d66aff0e48deb5513001f1d81963 P 22b0b1ae6c504301ad5170c6bdcd179e: Time spent starting tablet: real 0.098s user 0.005s sys 0.001s
I20240503 02:11:38.956390 25909 raft_consensus.cc:686] T 4ce4d66aff0e48deb5513001f1d81963 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 LEADER]: Becoming Leader. State: Replica: 22b0b1ae6c504301ad5170c6bdcd179e, State: Running, Role: LEADER
I20240503 02:11:38.956935 25909 consensus_queue.cc:237] T 4ce4d66aff0e48deb5513001f1d81963 P 22b0b1ae6c504301ad5170c6bdcd179e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } }
I20240503 02:11:38.959177 24847 catalog_manager.cc:5484] T 4ce4d66aff0e48deb5513001f1d81963 P 22b0b1ae6c504301ad5170c6bdcd179e reported cstate change: term changed from 0 to 1, leader changed from <none> to 22b0b1ae6c504301ad5170c6bdcd179e (127.22.108.2). New cstate: current_term: 1 leader_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } }
W20240503 02:11:39.357914 25918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:39.358223 25918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:39.365777 25918 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 118.904 ms
time per row: 0.0594521 ms
I20240503 02:11:39.621582 24848 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44436:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:11:39.629392 24968 tablet_service.cc:1467] Processing CreateTablet for tablet ed43f867d21d4bee9efc0b9bf4ee90c1 (DEFAULT_TABLE table=rebalance_test_table_2 [id=7aa88098eef344a7aebdefe66f69aa8b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:39.629392 24970 tablet_service.cc:1467] Processing CreateTablet for tablet 7c0b16db9b77442b922e3b7bf1247878 (DEFAULT_TABLE table=rebalance_test_table_2 [id=7aa88098eef344a7aebdefe66f69aa8b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:39.629612 24969 tablet_service.cc:1467] Processing CreateTablet for tablet 88000b61106c4087948a9aa37ecc297f (DEFAULT_TABLE table=rebalance_test_table_2 [id=7aa88098eef344a7aebdefe66f69aa8b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:39.629958 24968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed43f867d21d4bee9efc0b9bf4ee90c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:39.630168 24969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88000b61106c4087948a9aa37ecc297f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:39.630340 24970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c0b16db9b77442b922e3b7bf1247878. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:39.635653 25959 tablet_bootstrap.cc:492] T 88000b61106c4087948a9aa37ecc297f P 308436a6f7c7456b99971c666d3bd049: Bootstrap starting.
I20240503 02:11:39.637382 25959 tablet_bootstrap.cc:654] T 88000b61106c4087948a9aa37ecc297f P 308436a6f7c7456b99971c666d3bd049: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:39.638936 25959 tablet_bootstrap.cc:492] T 88000b61106c4087948a9aa37ecc297f P 308436a6f7c7456b99971c666d3bd049: No bootstrap required, opened a new log
I20240503 02:11:39.639050 25959 ts_tablet_manager.cc:1367] T 88000b61106c4087948a9aa37ecc297f P 308436a6f7c7456b99971c666d3bd049: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:11:39.639657 25959 raft_consensus.cc:348] T 88000b61106c4087948a9aa37ecc297f P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:39.639796 25959 raft_consensus.cc:374] T 88000b61106c4087948a9aa37ecc297f P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:39.639878 25959 raft_consensus.cc:729] T 88000b61106c4087948a9aa37ecc297f P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 308436a6f7c7456b99971c666d3bd049, State: Initialized, Role: FOLLOWER
I20240503 02:11:39.640095 25959 consensus_queue.cc:260] T 88000b61106c4087948a9aa37ecc297f P 308436a6f7c7456b99971c666d3bd049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:39.640287 25959 raft_consensus.cc:388] T 88000b61106c4087948a9aa37ecc297f P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:39.640363 25959 raft_consensus.cc:482] T 88000b61106c4087948a9aa37ecc297f P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:39.640487 25959 raft_consensus.cc:3037] T 88000b61106c4087948a9aa37ecc297f P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:39.642488 25959 raft_consensus.cc:504] T 88000b61106c4087948a9aa37ecc297f P 308436a6f7c7456b99971c666d3bd049 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:39.642714 25959 leader_election.cc:302] T 88000b61106c4087948a9aa37ecc297f P 308436a6f7c7456b99971c666d3bd049 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 308436a6f7c7456b99971c666d3bd049; no voters:
I20240503 02:11:39.643095 25959 leader_election.cc:288] T 88000b61106c4087948a9aa37ecc297f P 308436a6f7c7456b99971c666d3bd049 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:39.643257 25961 raft_consensus.cc:2781] T 88000b61106c4087948a9aa37ecc297f P 308436a6f7c7456b99971c666d3bd049 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:39.643945 25961 raft_consensus.cc:686] T 88000b61106c4087948a9aa37ecc297f P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Becoming Leader. State: Replica: 308436a6f7c7456b99971c666d3bd049, State: Running, Role: LEADER
I20240503 02:11:39.644033 25959 ts_tablet_manager.cc:1398] T 88000b61106c4087948a9aa37ecc297f P 308436a6f7c7456b99971c666d3bd049: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 02:11:39.644264 25959 tablet_bootstrap.cc:492] T 7c0b16db9b77442b922e3b7bf1247878 P 308436a6f7c7456b99971c666d3bd049: Bootstrap starting.
I20240503 02:11:39.644219 25961 consensus_queue.cc:237] T 88000b61106c4087948a9aa37ecc297f P 308436a6f7c7456b99971c666d3bd049 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:39.645761 25959 tablet_bootstrap.cc:654] T 7c0b16db9b77442b922e3b7bf1247878 P 308436a6f7c7456b99971c666d3bd049: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:39.647549 25959 tablet_bootstrap.cc:492] T 7c0b16db9b77442b922e3b7bf1247878 P 308436a6f7c7456b99971c666d3bd049: No bootstrap required, opened a new log
I20240503 02:11:39.647706 25959 ts_tablet_manager.cc:1367] T 7c0b16db9b77442b922e3b7bf1247878 P 308436a6f7c7456b99971c666d3bd049: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:11:39.648267 25959 raft_consensus.cc:348] T 7c0b16db9b77442b922e3b7bf1247878 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:39.648424 25959 raft_consensus.cc:374] T 7c0b16db9b77442b922e3b7bf1247878 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:39.648495 25959 raft_consensus.cc:729] T 7c0b16db9b77442b922e3b7bf1247878 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 308436a6f7c7456b99971c666d3bd049, State: Initialized, Role: FOLLOWER
I20240503 02:11:39.648322 24848 catalog_manager.cc:5484] T 88000b61106c4087948a9aa37ecc297f P 308436a6f7c7456b99971c666d3bd049 reported cstate change: term changed from 0 to 1, leader changed from <none> to 308436a6f7c7456b99971c666d3bd049 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "308436a6f7c7456b99971c666d3bd049" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } }
I20240503 02:11:39.648715 25959 consensus_queue.cc:260] T 7c0b16db9b77442b922e3b7bf1247878 P 308436a6f7c7456b99971c666d3bd049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:39.648842 25959 raft_consensus.cc:388] T 7c0b16db9b77442b922e3b7bf1247878 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:39.648932 25959 raft_consensus.cc:482] T 7c0b16db9b77442b922e3b7bf1247878 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:39.649019 25959 raft_consensus.cc:3037] T 7c0b16db9b77442b922e3b7bf1247878 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:39.651430 25959 raft_consensus.cc:504] T 7c0b16db9b77442b922e3b7bf1247878 P 308436a6f7c7456b99971c666d3bd049 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:39.651615 25959 leader_election.cc:302] T 7c0b16db9b77442b922e3b7bf1247878 P 308436a6f7c7456b99971c666d3bd049 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 308436a6f7c7456b99971c666d3bd049; no voters:
I20240503 02:11:39.651787 25959 leader_election.cc:288] T 7c0b16db9b77442b922e3b7bf1247878 P 308436a6f7c7456b99971c666d3bd049 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:39.651850 25961 raft_consensus.cc:2781] T 7c0b16db9b77442b922e3b7bf1247878 P 308436a6f7c7456b99971c666d3bd049 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:39.651993 25961 raft_consensus.cc:686] T 7c0b16db9b77442b922e3b7bf1247878 P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Becoming Leader. State: Replica: 308436a6f7c7456b99971c666d3bd049, State: Running, Role: LEADER
I20240503 02:11:39.652220 25959 ts_tablet_manager.cc:1398] T 7c0b16db9b77442b922e3b7bf1247878 P 308436a6f7c7456b99971c666d3bd049: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:11:39.652246 25961 consensus_queue.cc:237] T 7c0b16db9b77442b922e3b7bf1247878 P 308436a6f7c7456b99971c666d3bd049 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:39.652526 25959 tablet_bootstrap.cc:492] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 308436a6f7c7456b99971c666d3bd049: Bootstrap starting.
I20240503 02:11:39.654037 25959 tablet_bootstrap.cc:654] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 308436a6f7c7456b99971c666d3bd049: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:39.654686 24848 catalog_manager.cc:5484] T 7c0b16db9b77442b922e3b7bf1247878 P 308436a6f7c7456b99971c666d3bd049 reported cstate change: term changed from 0 to 1, leader changed from <none> to 308436a6f7c7456b99971c666d3bd049 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "308436a6f7c7456b99971c666d3bd049" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } }
I20240503 02:11:39.655889 25959 tablet_bootstrap.cc:492] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 308436a6f7c7456b99971c666d3bd049: No bootstrap required, opened a new log
I20240503 02:11:39.656021 25959 ts_tablet_manager.cc:1367] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 308436a6f7c7456b99971c666d3bd049: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:11:39.656800 25959 raft_consensus.cc:348] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:39.657002 25959 raft_consensus.cc:374] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:39.657181 25959 raft_consensus.cc:729] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 308436a6f7c7456b99971c666d3bd049, State: Initialized, Role: FOLLOWER
I20240503 02:11:39.657511 25959 consensus_queue.cc:260] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 308436a6f7c7456b99971c666d3bd049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:39.657769 25959 raft_consensus.cc:388] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:39.657900 25959 raft_consensus.cc:482] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:39.658020 25959 raft_consensus.cc:3037] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:39.659960 25959 raft_consensus.cc:504] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 308436a6f7c7456b99971c666d3bd049 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:39.660192 25959 leader_election.cc:302] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 308436a6f7c7456b99971c666d3bd049 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 308436a6f7c7456b99971c666d3bd049; no voters:
I20240503 02:11:39.660362 25959 leader_election.cc:288] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 308436a6f7c7456b99971c666d3bd049 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:39.660444 25962 raft_consensus.cc:2781] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 308436a6f7c7456b99971c666d3bd049 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:39.660933 25962 raft_consensus.cc:686] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Becoming Leader. State: Replica: 308436a6f7c7456b99971c666d3bd049, State: Running, Role: LEADER
I20240503 02:11:39.661239 25962 consensus_queue.cc:237] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 308436a6f7c7456b99971c666d3bd049 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:39.663689 24848 catalog_manager.cc:5484] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 308436a6f7c7456b99971c666d3bd049 reported cstate change: term changed from 0 to 1, leader changed from <none> to 308436a6f7c7456b99971c666d3bd049 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "308436a6f7c7456b99971c666d3bd049" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } }
I20240503 02:11:39.666934 25959 ts_tablet_manager.cc:1398] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 308436a6f7c7456b99971c666d3bd049: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
W20240503 02:11:39.997769 25973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:39.998087 25973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:40.005751 25973 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 118.431 ms
time per row: 0.0592156 ms
I20240503 02:11:40.260555 24848 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44436:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:11:40.269119 25099 tablet_service.cc:1467] Processing CreateTablet for tablet 7b671abb263d4848b7b909927b7df380 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ac074aff80ae4832bb677cde4982aed7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:40.269193 25101 tablet_service.cc:1467] Processing CreateTablet for tablet af70ad6a7a8a4051beb408ead22d30e3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ac074aff80ae4832bb677cde4982aed7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:40.269232 25100 tablet_service.cc:1467] Processing CreateTablet for tablet 4e5d560763774c0cba004753269a158c (DEFAULT_TABLE table=rebalance_test_table_3 [id=ac074aff80ae4832bb677cde4982aed7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:40.269639 25099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b671abb263d4848b7b909927b7df380. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:40.269807 25100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e5d560763774c0cba004753269a158c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:40.270167 25101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af70ad6a7a8a4051beb408ead22d30e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:40.275656 26009 tablet_bootstrap.cc:492] T af70ad6a7a8a4051beb408ead22d30e3 P 22b0b1ae6c504301ad5170c6bdcd179e: Bootstrap starting.
I20240503 02:11:40.277298 26009 tablet_bootstrap.cc:654] T af70ad6a7a8a4051beb408ead22d30e3 P 22b0b1ae6c504301ad5170c6bdcd179e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:40.278848 26009 tablet_bootstrap.cc:492] T af70ad6a7a8a4051beb408ead22d30e3 P 22b0b1ae6c504301ad5170c6bdcd179e: No bootstrap required, opened a new log
I20240503 02:11:40.278965 26009 ts_tablet_manager.cc:1367] T af70ad6a7a8a4051beb408ead22d30e3 P 22b0b1ae6c504301ad5170c6bdcd179e: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:11:40.279572 26009 raft_consensus.cc:348] T af70ad6a7a8a4051beb408ead22d30e3 P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } }
I20240503 02:11:40.279719 26009 raft_consensus.cc:374] T af70ad6a7a8a4051beb408ead22d30e3 P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:40.279788 26009 raft_consensus.cc:729] T af70ad6a7a8a4051beb408ead22d30e3 P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b0b1ae6c504301ad5170c6bdcd179e, State: Initialized, Role: FOLLOWER
I20240503 02:11:40.280015 26009 consensus_queue.cc:260] T af70ad6a7a8a4051beb408ead22d30e3 P 22b0b1ae6c504301ad5170c6bdcd179e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } }
I20240503 02:11:40.280195 26009 raft_consensus.cc:388] T af70ad6a7a8a4051beb408ead22d30e3 P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:40.280318 26009 raft_consensus.cc:482] T af70ad6a7a8a4051beb408ead22d30e3 P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:40.280426 26009 raft_consensus.cc:3037] T af70ad6a7a8a4051beb408ead22d30e3 P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:40.282287 26009 raft_consensus.cc:504] T af70ad6a7a8a4051beb408ead22d30e3 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } }
I20240503 02:11:40.282503 26009 leader_election.cc:302] T af70ad6a7a8a4051beb408ead22d30e3 P 22b0b1ae6c504301ad5170c6bdcd179e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22b0b1ae6c504301ad5170c6bdcd179e; no voters:
I20240503 02:11:40.282902 26009 leader_election.cc:288] T af70ad6a7a8a4051beb408ead22d30e3 P 22b0b1ae6c504301ad5170c6bdcd179e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:40.283077 26011 raft_consensus.cc:2781] T af70ad6a7a8a4051beb408ead22d30e3 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:40.283584 26009 ts_tablet_manager.cc:1398] T af70ad6a7a8a4051beb408ead22d30e3 P 22b0b1ae6c504301ad5170c6bdcd179e: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240503 02:11:40.283835 26009 tablet_bootstrap.cc:492] T 7b671abb263d4848b7b909927b7df380 P 22b0b1ae6c504301ad5170c6bdcd179e: Bootstrap starting.
I20240503 02:11:40.283955 26011 raft_consensus.cc:686] T af70ad6a7a8a4051beb408ead22d30e3 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 LEADER]: Becoming Leader. State: Replica: 22b0b1ae6c504301ad5170c6bdcd179e, State: Running, Role: LEADER
I20240503 02:11:40.284247 26011 consensus_queue.cc:237] T af70ad6a7a8a4051beb408ead22d30e3 P 22b0b1ae6c504301ad5170c6bdcd179e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } }
I20240503 02:11:40.285485 26009 tablet_bootstrap.cc:654] T 7b671abb263d4848b7b909927b7df380 P 22b0b1ae6c504301ad5170c6bdcd179e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:40.287101 24848 catalog_manager.cc:5484] T af70ad6a7a8a4051beb408ead22d30e3 P 22b0b1ae6c504301ad5170c6bdcd179e reported cstate change: term changed from 0 to 1, leader changed from <none> to 22b0b1ae6c504301ad5170c6bdcd179e (127.22.108.2). New cstate: current_term: 1 leader_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } }
I20240503 02:11:40.287447 26009 tablet_bootstrap.cc:492] T 7b671abb263d4848b7b909927b7df380 P 22b0b1ae6c504301ad5170c6bdcd179e: No bootstrap required, opened a new log
I20240503 02:11:40.287593 26009 ts_tablet_manager.cc:1367] T 7b671abb263d4848b7b909927b7df380 P 22b0b1ae6c504301ad5170c6bdcd179e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:11:40.288139 26009 raft_consensus.cc:348] T 7b671abb263d4848b7b909927b7df380 P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } }
I20240503 02:11:40.288301 26009 raft_consensus.cc:374] T 7b671abb263d4848b7b909927b7df380 P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:40.288384 26009 raft_consensus.cc:729] T 7b671abb263d4848b7b909927b7df380 P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b0b1ae6c504301ad5170c6bdcd179e, State: Initialized, Role: FOLLOWER
I20240503 02:11:40.288620 26009 consensus_queue.cc:260] T 7b671abb263d4848b7b909927b7df380 P 22b0b1ae6c504301ad5170c6bdcd179e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } }
I20240503 02:11:40.288758 26009 raft_consensus.cc:388] T 7b671abb263d4848b7b909927b7df380 P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:40.288856 26009 raft_consensus.cc:482] T 7b671abb263d4848b7b909927b7df380 P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:40.288954 26009 raft_consensus.cc:3037] T 7b671abb263d4848b7b909927b7df380 P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:40.291241 26009 raft_consensus.cc:504] T 7b671abb263d4848b7b909927b7df380 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } }
I20240503 02:11:40.291496 26009 leader_election.cc:302] T 7b671abb263d4848b7b909927b7df380 P 22b0b1ae6c504301ad5170c6bdcd179e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22b0b1ae6c504301ad5170c6bdcd179e; no voters:
I20240503 02:11:40.291733 26009 leader_election.cc:288] T 7b671abb263d4848b7b909927b7df380 P 22b0b1ae6c504301ad5170c6bdcd179e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:40.292073 26013 raft_consensus.cc:2781] T 7b671abb263d4848b7b909927b7df380 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:40.292217 26009 ts_tablet_manager.cc:1398] T 7b671abb263d4848b7b909927b7df380 P 22b0b1ae6c504301ad5170c6bdcd179e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 02:11:40.292224 26013 raft_consensus.cc:686] T 7b671abb263d4848b7b909927b7df380 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 LEADER]: Becoming Leader. State: Replica: 22b0b1ae6c504301ad5170c6bdcd179e, State: Running, Role: LEADER
I20240503 02:11:40.292485 26009 tablet_bootstrap.cc:492] T 4e5d560763774c0cba004753269a158c P 22b0b1ae6c504301ad5170c6bdcd179e: Bootstrap starting.
I20240503 02:11:40.292572 26013 consensus_queue.cc:237] T 7b671abb263d4848b7b909927b7df380 P 22b0b1ae6c504301ad5170c6bdcd179e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } }
I20240503 02:11:40.293809 26009 tablet_bootstrap.cc:654] T 4e5d560763774c0cba004753269a158c P 22b0b1ae6c504301ad5170c6bdcd179e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:40.294692 24843 catalog_manager.cc:5484] T 7b671abb263d4848b7b909927b7df380 P 22b0b1ae6c504301ad5170c6bdcd179e reported cstate change: term changed from 0 to 1, leader changed from <none> to 22b0b1ae6c504301ad5170c6bdcd179e (127.22.108.2). New cstate: current_term: 1 leader_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } }
I20240503 02:11:40.295675 26009 tablet_bootstrap.cc:492] T 4e5d560763774c0cba004753269a158c P 22b0b1ae6c504301ad5170c6bdcd179e: No bootstrap required, opened a new log
I20240503 02:11:40.295778 26009 ts_tablet_manager.cc:1367] T 4e5d560763774c0cba004753269a158c P 22b0b1ae6c504301ad5170c6bdcd179e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:11:40.296329 26009 raft_consensus.cc:348] T 4e5d560763774c0cba004753269a158c P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } }
I20240503 02:11:40.296489 26009 raft_consensus.cc:374] T 4e5d560763774c0cba004753269a158c P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:40.296586 26009 raft_consensus.cc:729] T 4e5d560763774c0cba004753269a158c P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b0b1ae6c504301ad5170c6bdcd179e, State: Initialized, Role: FOLLOWER
I20240503 02:11:40.296797 26009 consensus_queue.cc:260] T 4e5d560763774c0cba004753269a158c P 22b0b1ae6c504301ad5170c6bdcd179e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } }
I20240503 02:11:40.296990 26009 raft_consensus.cc:388] T 4e5d560763774c0cba004753269a158c P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:40.297135 26009 raft_consensus.cc:482] T 4e5d560763774c0cba004753269a158c P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:40.297262 26009 raft_consensus.cc:3037] T 4e5d560763774c0cba004753269a158c P 22b0b1ae6c504301ad5170c6bdcd179e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:40.299111 26009 raft_consensus.cc:504] T 4e5d560763774c0cba004753269a158c P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } }
I20240503 02:11:40.299332 26009 leader_election.cc:302] T 4e5d560763774c0cba004753269a158c P 22b0b1ae6c504301ad5170c6bdcd179e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22b0b1ae6c504301ad5170c6bdcd179e; no voters:
I20240503 02:11:40.299502 26009 leader_election.cc:288] T 4e5d560763774c0cba004753269a158c P 22b0b1ae6c504301ad5170c6bdcd179e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:40.299580 26011 raft_consensus.cc:2781] T 4e5d560763774c0cba004753269a158c P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:40.299778 26011 raft_consensus.cc:686] T 4e5d560763774c0cba004753269a158c P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 LEADER]: Becoming Leader. State: Replica: 22b0b1ae6c504301ad5170c6bdcd179e, State: Running, Role: LEADER
I20240503 02:11:40.299988 26009 ts_tablet_manager.cc:1398] T 4e5d560763774c0cba004753269a158c P 22b0b1ae6c504301ad5170c6bdcd179e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:11:40.300025 26011 consensus_queue.cc:237] T 4e5d560763774c0cba004753269a158c P 22b0b1ae6c504301ad5170c6bdcd179e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } }
I20240503 02:11:40.301940 24843 catalog_manager.cc:5484] T 4e5d560763774c0cba004753269a158c P 22b0b1ae6c504301ad5170c6bdcd179e reported cstate change: term changed from 0 to 1, leader changed from <none> to 22b0b1ae6c504301ad5170c6bdcd179e (127.22.108.2). New cstate: current_term: 1 leader_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } }
W20240503 02:11:40.637717 26020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:40.638078 26020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:40.645807 26020 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 129.465 ms
time per row: 0.0647325 ms
I20240503 02:11:40.912580 24848 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44436:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:11:40.921718 24968 tablet_service.cc:1467] Processing CreateTablet for tablet fb78606eb2e74e19ab30bf7a119ea7e6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=83cdd3376f31411b849eb85a2660ae9f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:40.921753 24970 tablet_service.cc:1467] Processing CreateTablet for tablet 923672905ffb452993c5dcf283f4a399 (DEFAULT_TABLE table=rebalance_test_table_4 [id=83cdd3376f31411b849eb85a2660ae9f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:40.922010 24969 tablet_service.cc:1467] Processing CreateTablet for tablet f6920bfd97624ad381fd72b688ab73f1 (DEFAULT_TABLE table=rebalance_test_table_4 [id=83cdd3376f31411b849eb85a2660ae9f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:40.922236 24968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb78606eb2e74e19ab30bf7a119ea7e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:40.922477 24969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6920bfd97624ad381fd72b688ab73f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:40.922796 24970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 923672905ffb452993c5dcf283f4a399. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:40.928346 26057 tablet_bootstrap.cc:492] T fb78606eb2e74e19ab30bf7a119ea7e6 P 308436a6f7c7456b99971c666d3bd049: Bootstrap starting.
I20240503 02:11:40.929774 26057 tablet_bootstrap.cc:654] T fb78606eb2e74e19ab30bf7a119ea7e6 P 308436a6f7c7456b99971c666d3bd049: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:40.931512 26057 tablet_bootstrap.cc:492] T fb78606eb2e74e19ab30bf7a119ea7e6 P 308436a6f7c7456b99971c666d3bd049: No bootstrap required, opened a new log
I20240503 02:11:40.931645 26057 ts_tablet_manager.cc:1367] T fb78606eb2e74e19ab30bf7a119ea7e6 P 308436a6f7c7456b99971c666d3bd049: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:11:40.932191 26057 raft_consensus.cc:348] T fb78606eb2e74e19ab30bf7a119ea7e6 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:40.932344 26057 raft_consensus.cc:374] T fb78606eb2e74e19ab30bf7a119ea7e6 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:40.932451 26057 raft_consensus.cc:729] T fb78606eb2e74e19ab30bf7a119ea7e6 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 308436a6f7c7456b99971c666d3bd049, State: Initialized, Role: FOLLOWER
I20240503 02:11:40.932657 26057 consensus_queue.cc:260] T fb78606eb2e74e19ab30bf7a119ea7e6 P 308436a6f7c7456b99971c666d3bd049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:40.932807 26057 raft_consensus.cc:388] T fb78606eb2e74e19ab30bf7a119ea7e6 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:40.932876 26057 raft_consensus.cc:482] T fb78606eb2e74e19ab30bf7a119ea7e6 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:40.932984 26057 raft_consensus.cc:3037] T fb78606eb2e74e19ab30bf7a119ea7e6 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:40.934691 26057 raft_consensus.cc:504] T fb78606eb2e74e19ab30bf7a119ea7e6 P 308436a6f7c7456b99971c666d3bd049 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:40.934938 26057 leader_election.cc:302] T fb78606eb2e74e19ab30bf7a119ea7e6 P 308436a6f7c7456b99971c666d3bd049 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 308436a6f7c7456b99971c666d3bd049; no voters:
I20240503 02:11:40.935344 26057 leader_election.cc:288] T fb78606eb2e74e19ab30bf7a119ea7e6 P 308436a6f7c7456b99971c666d3bd049 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:40.935508 26059 raft_consensus.cc:2781] T fb78606eb2e74e19ab30bf7a119ea7e6 P 308436a6f7c7456b99971c666d3bd049 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:40.936038 26059 raft_consensus.cc:686] T fb78606eb2e74e19ab30bf7a119ea7e6 P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Becoming Leader. State: Replica: 308436a6f7c7456b99971c666d3bd049, State: Running, Role: LEADER
I20240503 02:11:40.936314 26059 consensus_queue.cc:237] T fb78606eb2e74e19ab30bf7a119ea7e6 P 308436a6f7c7456b99971c666d3bd049 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:40.936419 26057 ts_tablet_manager.cc:1398] T fb78606eb2e74e19ab30bf7a119ea7e6 P 308436a6f7c7456b99971c666d3bd049: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 02:11:40.936640 26057 tablet_bootstrap.cc:492] T f6920bfd97624ad381fd72b688ab73f1 P 308436a6f7c7456b99971c666d3bd049: Bootstrap starting.
I20240503 02:11:40.938340 26057 tablet_bootstrap.cc:654] T f6920bfd97624ad381fd72b688ab73f1 P 308436a6f7c7456b99971c666d3bd049: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:40.938795 24848 catalog_manager.cc:5484] T fb78606eb2e74e19ab30bf7a119ea7e6 P 308436a6f7c7456b99971c666d3bd049 reported cstate change: term changed from 0 to 1, leader changed from <none> to 308436a6f7c7456b99971c666d3bd049 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "308436a6f7c7456b99971c666d3bd049" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } }
I20240503 02:11:40.939843 26057 tablet_bootstrap.cc:492] T f6920bfd97624ad381fd72b688ab73f1 P 308436a6f7c7456b99971c666d3bd049: No bootstrap required, opened a new log
I20240503 02:11:40.939993 26057 ts_tablet_manager.cc:1367] T f6920bfd97624ad381fd72b688ab73f1 P 308436a6f7c7456b99971c666d3bd049: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:11:40.940770 26057 raft_consensus.cc:348] T f6920bfd97624ad381fd72b688ab73f1 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:40.940929 26057 raft_consensus.cc:374] T f6920bfd97624ad381fd72b688ab73f1 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:40.941006 26057 raft_consensus.cc:729] T f6920bfd97624ad381fd72b688ab73f1 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 308436a6f7c7456b99971c666d3bd049, State: Initialized, Role: FOLLOWER
I20240503 02:11:40.941262 26057 consensus_queue.cc:260] T f6920bfd97624ad381fd72b688ab73f1 P 308436a6f7c7456b99971c666d3bd049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:40.941442 26057 raft_consensus.cc:388] T f6920bfd97624ad381fd72b688ab73f1 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:40.941536 26057 raft_consensus.cc:482] T f6920bfd97624ad381fd72b688ab73f1 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:40.941627 26057 raft_consensus.cc:3037] T f6920bfd97624ad381fd72b688ab73f1 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:40.943781 26057 raft_consensus.cc:504] T f6920bfd97624ad381fd72b688ab73f1 P 308436a6f7c7456b99971c666d3bd049 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:40.944012 26057 leader_election.cc:302] T f6920bfd97624ad381fd72b688ab73f1 P 308436a6f7c7456b99971c666d3bd049 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 308436a6f7c7456b99971c666d3bd049; no voters:
I20240503 02:11:40.944190 26057 leader_election.cc:288] T f6920bfd97624ad381fd72b688ab73f1 P 308436a6f7c7456b99971c666d3bd049 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:40.944233 26059 raft_consensus.cc:2781] T f6920bfd97624ad381fd72b688ab73f1 P 308436a6f7c7456b99971c666d3bd049 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:40.944453 26059 raft_consensus.cc:686] T f6920bfd97624ad381fd72b688ab73f1 P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Becoming Leader. State: Replica: 308436a6f7c7456b99971c666d3bd049, State: Running, Role: LEADER
I20240503 02:11:40.944633 26057 ts_tablet_manager.cc:1398] T f6920bfd97624ad381fd72b688ab73f1 P 308436a6f7c7456b99971c666d3bd049: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:11:40.944667 26059 consensus_queue.cc:237] T f6920bfd97624ad381fd72b688ab73f1 P 308436a6f7c7456b99971c666d3bd049 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:40.944895 26057 tablet_bootstrap.cc:492] T 923672905ffb452993c5dcf283f4a399 P 308436a6f7c7456b99971c666d3bd049: Bootstrap starting.
I20240503 02:11:40.946660 26057 tablet_bootstrap.cc:654] T 923672905ffb452993c5dcf283f4a399 P 308436a6f7c7456b99971c666d3bd049: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:40.947093 24848 catalog_manager.cc:5484] T f6920bfd97624ad381fd72b688ab73f1 P 308436a6f7c7456b99971c666d3bd049 reported cstate change: term changed from 0 to 1, leader changed from <none> to 308436a6f7c7456b99971c666d3bd049 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "308436a6f7c7456b99971c666d3bd049" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } }
I20240503 02:11:40.948307 26057 tablet_bootstrap.cc:492] T 923672905ffb452993c5dcf283f4a399 P 308436a6f7c7456b99971c666d3bd049: No bootstrap required, opened a new log
I20240503 02:11:40.948452 26057 ts_tablet_manager.cc:1367] T 923672905ffb452993c5dcf283f4a399 P 308436a6f7c7456b99971c666d3bd049: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:11:40.949169 26057 raft_consensus.cc:348] T 923672905ffb452993c5dcf283f4a399 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:40.949343 26057 raft_consensus.cc:374] T 923672905ffb452993c5dcf283f4a399 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:40.949491 26057 raft_consensus.cc:729] T 923672905ffb452993c5dcf283f4a399 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 308436a6f7c7456b99971c666d3bd049, State: Initialized, Role: FOLLOWER
I20240503 02:11:40.949735 26057 consensus_queue.cc:260] T 923672905ffb452993c5dcf283f4a399 P 308436a6f7c7456b99971c666d3bd049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:40.949898 26057 raft_consensus.cc:388] T 923672905ffb452993c5dcf283f4a399 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:40.949972 26057 raft_consensus.cc:482] T 923672905ffb452993c5dcf283f4a399 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:40.950088 26057 raft_consensus.cc:3037] T 923672905ffb452993c5dcf283f4a399 P 308436a6f7c7456b99971c666d3bd049 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:40.951856 26057 raft_consensus.cc:504] T 923672905ffb452993c5dcf283f4a399 P 308436a6f7c7456b99971c666d3bd049 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:40.952065 26057 leader_election.cc:302] T 923672905ffb452993c5dcf283f4a399 P 308436a6f7c7456b99971c666d3bd049 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 308436a6f7c7456b99971c666d3bd049; no voters:
I20240503 02:11:40.952231 26057 leader_election.cc:288] T 923672905ffb452993c5dcf283f4a399 P 308436a6f7c7456b99971c666d3bd049 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:40.952349 26059 raft_consensus.cc:2781] T 923672905ffb452993c5dcf283f4a399 P 308436a6f7c7456b99971c666d3bd049 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:40.952534 26059 raft_consensus.cc:686] T 923672905ffb452993c5dcf283f4a399 P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Becoming Leader. State: Replica: 308436a6f7c7456b99971c666d3bd049, State: Running, Role: LEADER
I20240503 02:11:40.952652 26057 ts_tablet_manager.cc:1398] T 923672905ffb452993c5dcf283f4a399 P 308436a6f7c7456b99971c666d3bd049: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:11:40.952801 26059 consensus_queue.cc:237] T 923672905ffb452993c5dcf283f4a399 P 308436a6f7c7456b99971c666d3bd049 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:40.954923 24848 catalog_manager.cc:5484] T 923672905ffb452993c5dcf283f4a399 P 308436a6f7c7456b99971c666d3bd049 reported cstate change: term changed from 0 to 1, leader changed from <none> to 308436a6f7c7456b99971c666d3bd049 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "308436a6f7c7456b99971c666d3bd049" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } }
W20240503 02:11:41.300753 26068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:41.301122 26068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:41.309664 26068 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 100.996 ms
time per row: 0.0504982 ms
I20240503 02:11:41.547686 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.3:36823
--local_ip_for_outbound_sockets=127.22.108.3
--tserver_master_addrs=127.22.108.62:38409
--webserver_port=36617
--webserver_interface=127.22.108.3
--builtin_ntp_servers=127.22.108.20:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 02:11:41.879721 26102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:41.880030 26102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:41.880215 26102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:41.887921 26102 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:11:41.888057 26102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:41.888211 26102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.3
I20240503 02:11:41.897110 26102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.3:36823
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.108.3
--webserver_port=36617
--tserver_master_addrs=127.22.108.62:38409
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.3
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:41.898476 26102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:41.900321 26102 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:41.913410 26110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:41.914206 26108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:41.914969 26107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:41.915961 26102 server_base.cc:1034] running on GCE node
I20240503 02:11:41.916538 26102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:41.917495 26102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:41.918684 26102 hybrid_clock.cc:648] HybridClock initialized: now 1714702301918644 us; error 44 us; skew 500 ppm
I20240503 02:11:41.918983 26102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:41.921223 26102 webserver.cc:460] Webserver started at http://127.22.108.3:36617/ using document root <none> and password file <none>
I20240503 02:11:41.921954 26102 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:41.922083 26102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:41.927208 26102 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20240503 02:11:41.929872 26116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:41.931103 26102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240503 02:11:41.931262 26102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b95ea99cb72f49c6b42c7e8e293cfc27"
format_stamp: "Formatted at 2024-05-03 02:11:31 on dist-test-slave-sc15"
I20240503 02:11:41.932073 26102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:41.945861 26102 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:41.946702 26102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:41.946982 26102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:41.947923 26102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:41.949507 26102 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:41.949589 26102 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:41.949721 26102 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:41.949779 26102 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:41.981966 26102 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.3:36823
I20240503 02:11:41.982000 26228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.3:36823 every 8 connection(s)
I20240503 02:11:41.983448 26102 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:11:41.986020 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 26102
I20240503 02:11:41.986642 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.4:45401
--local_ip_for_outbound_sockets=127.22.108.4
--tserver_master_addrs=127.22.108.62:38409
--webserver_port=45785
--webserver_interface=127.22.108.4
--builtin_ntp_servers=127.22.108.20:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:11:41.998616 26229 heartbeater.cc:347] Connected to a master server at 127.22.108.62:38409
I20240503 02:11:41.998925 26229 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:41.999778 26229 heartbeater.cc:511] Master 127.22.108.62:38409 requested a full tablet report, sending...
I20240503 02:11:42.001607 24848 ts_manager.cc:194] Re-registered known tserver with Master: b95ea99cb72f49c6b42c7e8e293cfc27 (127.22.108.3:36823)
I20240503 02:11:42.002872 24848 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.3:55009
W20240503 02:11:42.323602 26233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:42.323913 26233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:42.324007 26233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:42.331367 26233 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:11:42.331514 26233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:42.331625 26233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.4
I20240503 02:11:42.340691 26233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.4:45401
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.108.4
--webserver_port=45785
--tserver_master_addrs=127.22.108.62:38409
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.4
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:42.342065 26233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:42.343868 26233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:42.356887 26239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:42.357857 26238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:42.358850 26241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:42.359664 26233 server_base.cc:1034] running on GCE node
I20240503 02:11:42.360270 26233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:42.361307 26233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:42.362542 26233 hybrid_clock.cc:648] HybridClock initialized: now 1714702302362497 us; error 43 us; skew 500 ppm
I20240503 02:11:42.362838 26233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:42.365192 26233 webserver.cc:460] Webserver started at http://127.22.108.4:45785/ using document root <none> and password file <none>
I20240503 02:11:42.365911 26233 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:42.366021 26233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:42.371147 26233 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20240503 02:11:42.373849 26247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:42.375186 26233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:11:42.375334 26233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "0163beb78bfd4d399a70ebacb8f9aa9c"
format_stamp: "Formatted at 2024-05-03 02:11:31 on dist-test-slave-sc15"
I20240503 02:11:42.376155 26233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:42.400188 26233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:42.401109 26233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:42.401427 26233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:42.402390 26233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:42.403977 26233 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:42.404047 26233 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:42.404165 26233 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:42.404234 26233 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:42.434805 26233 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.4:45401
I20240503 02:11:42.434835 26359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.4:45401 every 8 connection(s)
I20240503 02:11:42.436383 26233 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:11:42.437726 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 26233
I20240503 02:11:42.438470 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.5:40741
--local_ip_for_outbound_sockets=127.22.108.5
--tserver_master_addrs=127.22.108.62:38409
--webserver_port=41907
--webserver_interface=127.22.108.5
--builtin_ntp_servers=127.22.108.20:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:11:42.449167 26360 heartbeater.cc:347] Connected to a master server at 127.22.108.62:38409
I20240503 02:11:42.449448 26360 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:42.450155 26360 heartbeater.cc:511] Master 127.22.108.62:38409 requested a full tablet report, sending...
I20240503 02:11:42.451474 24848 ts_manager.cc:194] Re-registered known tserver with Master: 0163beb78bfd4d399a70ebacb8f9aa9c (127.22.108.4:45401)
I20240503 02:11:42.452263 24848 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.4:52989
W20240503 02:11:42.769513 26363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:42.769829 26363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:42.769969 26363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:42.777535 26363 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:11:42.777680 26363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:42.777838 26363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.5
I20240503 02:11:42.786971 26363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.5:40741
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.108.5
--webserver_port=41907
--tserver_master_addrs=127.22.108.62:38409
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.5
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:42.788277 26363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:42.790072 26363 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:42.802517 26369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:42.803225 26368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:42.803941 26371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:42.805132 26363 server_base.cc:1034] running on GCE node
I20240503 02:11:42.805739 26363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:42.806701 26363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:42.807915 26363 hybrid_clock.cc:648] HybridClock initialized: now 1714702302807853 us; error 54 us; skew 500 ppm
I20240503 02:11:42.808260 26363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:42.810652 26363 webserver.cc:460] Webserver started at http://127.22.108.5:41907/ using document root <none> and password file <none>
I20240503 02:11:42.811398 26363 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:42.811506 26363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:42.816536 26363 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20240503 02:11:42.819113 26377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:42.820360 26363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 02:11:42.820494 26363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "1b2a71b48d4e40679504230af08279e9"
format_stamp: "Formatted at 2024-05-03 02:11:32 on dist-test-slave-sc15"
I20240503 02:11:42.821303 26363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:42.839033 26363 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:42.839802 26363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:42.840061 26363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:42.840898 26363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:42.842386 26363 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:42.842455 26363 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:42.842543 26363 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:42.842586 26363 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:42.874348 26363 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.5:40741
I20240503 02:11:42.874379 26489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.5:40741 every 8 connection(s)
I20240503 02:11:42.875856 26363 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:11:42.877416 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 26363
I20240503 02:11:42.878141 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.6:43731
--local_ip_for_outbound_sockets=127.22.108.6
--tserver_master_addrs=127.22.108.62:38409
--webserver_port=35125
--webserver_interface=127.22.108.6
--builtin_ntp_servers=127.22.108.20:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:11:42.890026 26490 heartbeater.cc:347] Connected to a master server at 127.22.108.62:38409
I20240503 02:11:42.890331 26490 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:42.891095 26490 heartbeater.cc:511] Master 127.22.108.62:38409 requested a full tablet report, sending...
I20240503 02:11:42.892462 24848 ts_manager.cc:194] Re-registered known tserver with Master: 1b2a71b48d4e40679504230af08279e9 (127.22.108.5:40741)
I20240503 02:11:42.893321 24848 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.5:57535
I20240503 02:11:43.005555 26229 heartbeater.cc:503] Master 127.22.108.62:38409 was elected leader, sending a full tablet report...
W20240503 02:11:43.211622 26493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:43.211922 26493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:43.212078 26493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:43.219504 26493 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:11:43.219655 26493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:43.219820 26493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.6
I20240503 02:11:43.229288 26493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.6:43731
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.22.108.6
--webserver_port=35125
--tserver_master_addrs=127.22.108.62:38409
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.6
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:43.230683 26493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:43.232463 26493 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:43.245258 26499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:43.246233 26498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:43.246896 26501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:43.247745 26493 server_base.cc:1034] running on GCE node
I20240503 02:11:43.248394 26493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:43.249440 26493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:43.250660 26493 hybrid_clock.cc:648] HybridClock initialized: now 1714702303250617 us; error 36 us; skew 500 ppm
I20240503 02:11:43.251015 26493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:43.253424 26493 webserver.cc:460] Webserver started at http://127.22.108.6:35125/ using document root <none> and password file <none>
I20240503 02:11:43.254174 26493 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:43.254310 26493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:43.259541 26493 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 02:11:43.262284 26507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:43.263593 26493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:11:43.263748 26493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6c704dfc19814db996a1c0e363e760ef"
format_stamp: "Formatted at 2024-05-03 02:11:32 on dist-test-slave-sc15"
I20240503 02:11:43.264601 26493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:43.288069 26493 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:43.288935 26493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:43.289249 26493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:43.290207 26493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:43.291754 26493 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:43.291836 26493 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:43.291972 26493 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:43.292028 26493 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:43.323930 26493 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.6:43731
I20240503 02:11:43.323962 26619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.6:43731 every 8 connection(s)
I20240503 02:11:43.325433 26493 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 02:11:43.328032 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 26493
I20240503 02:11:43.328696 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.7:34851
--local_ip_for_outbound_sockets=127.22.108.7
--tserver_master_addrs=127.22.108.62:38409
--webserver_port=45859
--webserver_interface=127.22.108.7
--builtin_ntp_servers=127.22.108.20:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:11:43.338519 26620 heartbeater.cc:347] Connected to a master server at 127.22.108.62:38409
I20240503 02:11:43.338864 26620 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:43.339785 26620 heartbeater.cc:511] Master 127.22.108.62:38409 requested a full tablet report, sending...
I20240503 02:11:43.341367 24848 ts_manager.cc:194] Re-registered known tserver with Master: 6c704dfc19814db996a1c0e363e760ef (127.22.108.6:43731)
I20240503 02:11:43.342221 24848 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.6:48789
I20240503 02:11:43.454651 26360 heartbeater.cc:503] Master 127.22.108.62:38409 was elected leader, sending a full tablet report...
W20240503 02:11:43.667534 26623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:43.667824 26623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:43.667906 26623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:43.675192 26623 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:11:43.675307 26623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:43.675421 26623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.7
I20240503 02:11:43.684542 26623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.7:34851
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.22.108.7
--webserver_port=45859
--tserver_master_addrs=127.22.108.62:38409
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.7
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:43.685804 26623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:43.687418 26623 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:43.699939 26628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:43.700917 26631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:43.701730 26629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:43.702530 26623 server_base.cc:1034] running on GCE node
I20240503 02:11:43.703135 26623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:43.704097 26623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:43.705288 26623 hybrid_clock.cc:648] HybridClock initialized: now 1714702303705236 us; error 80 us; skew 500 ppm
I20240503 02:11:43.705597 26623 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:43.712378 26623 webserver.cc:460] Webserver started at http://127.22.108.7:45859/ using document root <none> and password file <none>
I20240503 02:11:43.713191 26623 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:43.713330 26623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:43.718573 26623 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:11:43.721288 26637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:43.722608 26623 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:11:43.722779 26623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "03afb03fe005462eb07ada37fe23c3cf"
format_stamp: "Formatted at 2024-05-03 02:11:33 on dist-test-slave-sc15"
I20240503 02:11:43.724118 26623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:43.738914 26623 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:43.739667 26623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:43.739953 26623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:43.740837 26623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:43.742354 26623 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:43.742432 26623 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:43.742560 26623 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:43.742617 26623 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:43.776955 26623 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.7:34851
I20240503 02:11:43.776971 26749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.7:34851 every 8 connection(s)
I20240503 02:11:43.778424 26623 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 02:11:43.788167 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 26623
I20240503 02:11:43.791267 26750 heartbeater.cc:347] Connected to a master server at 127.22.108.62:38409
I20240503 02:11:43.791600 26750 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:43.792615 26750 heartbeater.cc:511] Master 127.22.108.62:38409 requested a full tablet report, sending...
I20240503 02:11:43.794100 24848 ts_manager.cc:194] Re-registered known tserver with Master: 03afb03fe005462eb07ada37fe23c3cf (127.22.108.7:34851)
I20240503 02:11:43.794831 24848 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.7:55159
I20240503 02:11:43.895892 26490 heartbeater.cc:503] Master 127.22.108.62:38409 was elected leader, sending a full tablet report...
I20240503 02:11:44.276948 26295 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:44.293280 26685 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:44.303541 26425 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:44.312089 25100 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:11:44.344177 24970 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:11:44.346609 26620 heartbeater.cc:503] Master 127.22.108.62:38409 was elected leader, sending a full tablet report...
I20240503 02:11:44.356891 26555 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:44.363389 26164 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:44.540140 26295 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:44.548134 26685 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:44.557494 26425 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:44.564824 25100 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:11:44.565454 24970 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:11:44.565532 26555 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:44.574133 26164 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:44.686602 25121 consensus_queue.cc:237] T 7b671abb263d4848b7b909927b7df380 P 22b0b1ae6c504301ad5170c6bdcd179e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "03afb03fe005462eb07ada37fe23c3cf" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 34851 } }
W20240503 02:11:44.695343 25055 consensus_peers.cc:485] T 7b671abb263d4848b7b909927b7df380 P 22b0b1ae6c504301ad5170c6bdcd179e -> Peer 03afb03fe005462eb07ada37fe23c3cf (127.22.108.7:34851): Couldn't send request to peer 03afb03fe005462eb07ada37fe23c3cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7b671abb263d4848b7b909927b7df380. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:44.797264 26750 heartbeater.cc:503] Master 127.22.108.62:38409 was elected leader, sending a full tablet report...
I20240503 02:11:45.242585 26851 ts_tablet_manager.cc:902] T 7b671abb263d4848b7b909927b7df380 P 03afb03fe005462eb07ada37fe23c3cf: Initiating tablet copy from peer 22b0b1ae6c504301ad5170c6bdcd179e (127.22.108.2:36213)
I20240503 02:11:45.244246 26851 tablet_copy_client.cc:308] T 7b671abb263d4848b7b909927b7df380 P 03afb03fe005462eb07ada37fe23c3cf: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.2:36213
I20240503 02:11:45.250111 25141 tablet_copy_service.cc:140] P 22b0b1ae6c504301ad5170c6bdcd179e: Received BeginTabletCopySession request for tablet 7b671abb263d4848b7b909927b7df380 from peer 03afb03fe005462eb07ada37fe23c3cf ({username='slave'} at 127.22.108.7:57651)
I20240503 02:11:45.250351 25141 tablet_copy_service.cc:161] P 22b0b1ae6c504301ad5170c6bdcd179e: Beginning new tablet copy session on tablet 7b671abb263d4848b7b909927b7df380 from peer 03afb03fe005462eb07ada37fe23c3cf at {username='slave'} at 127.22.108.7:57651: session id = 03afb03fe005462eb07ada37fe23c3cf-7b671abb263d4848b7b909927b7df380
I20240503 02:11:45.252895 25141 tablet_copy_source_session.cc:205] T 7b671abb263d4848b7b909927b7df380 P 22b0b1ae6c504301ad5170c6bdcd179e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:11:45.257164 26851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b671abb263d4848b7b909927b7df380. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:45.267163 26851 tablet_copy_client.cc:783] T 7b671abb263d4848b7b909927b7df380 P 03afb03fe005462eb07ada37fe23c3cf: tablet copy: Starting download of 0 data blocks...
I20240503 02:11:45.267452 26851 tablet_copy_client.cc:647] T 7b671abb263d4848b7b909927b7df380 P 03afb03fe005462eb07ada37fe23c3cf: tablet copy: Starting download of 1 WAL segments...
I20240503 02:11:45.269692 26851 tablet_copy_client.cc:523] T 7b671abb263d4848b7b909927b7df380 P 03afb03fe005462eb07ada37fe23c3cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:11:45.272392 26851 tablet_bootstrap.cc:492] T 7b671abb263d4848b7b909927b7df380 P 03afb03fe005462eb07ada37fe23c3cf: Bootstrap starting.
I20240503 02:11:45.281656 26851 log.cc:824] T 7b671abb263d4848b7b909927b7df380 P 03afb03fe005462eb07ada37fe23c3cf: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:45.312295 26851 tablet_bootstrap.cc:492] T 7b671abb263d4848b7b909927b7df380 P 03afb03fe005462eb07ada37fe23c3cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 02:11:45.312855 26851 tablet_bootstrap.cc:492] T 7b671abb263d4848b7b909927b7df380 P 03afb03fe005462eb07ada37fe23c3cf: Bootstrap complete.
I20240503 02:11:45.313302 26851 ts_tablet_manager.cc:1367] T 7b671abb263d4848b7b909927b7df380 P 03afb03fe005462eb07ada37fe23c3cf: Time spent bootstrapping tablet: real 0.041s user 0.026s sys 0.017s
I20240503 02:11:45.317819 26851 raft_consensus.cc:348] T 7b671abb263d4848b7b909927b7df380 P 03afb03fe005462eb07ada37fe23c3cf [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } }
I20240503 02:11:45.318634 26851 raft_consensus.cc:729] T 7b671abb263d4848b7b909927b7df380 P 03afb03fe005462eb07ada37fe23c3cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03afb03fe005462eb07ada37fe23c3cf, State: Initialized, Role: FOLLOWER
I20240503 02:11:45.319038 26851 consensus_queue.cc:260] T 7b671abb263d4848b7b909927b7df380 P 03afb03fe005462eb07ada37fe23c3cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "03afb03fe005462eb07ada37fe23c3cf" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 34851 } }
I20240503 02:11:45.320317 26851 ts_tablet_manager.cc:1398] T 7b671abb263d4848b7b909927b7df380 P 03afb03fe005462eb07ada37fe23c3cf: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20240503 02:11:45.321266 25141 tablet_copy_service.cc:342] P 22b0b1ae6c504301ad5170c6bdcd179e: Request end of tablet copy session 03afb03fe005462eb07ada37fe23c3cf-7b671abb263d4848b7b909927b7df380 received from {username='slave'} at 127.22.108.7:57651
I20240503 02:11:45.321414 25141 tablet_copy_service.cc:433] P 22b0b1ae6c504301ad5170c6bdcd179e: ending tablet copy session 03afb03fe005462eb07ada37fe23c3cf-7b671abb263d4848b7b909927b7df380 on tablet 7b671abb263d4848b7b909927b7df380 with peer 03afb03fe005462eb07ada37fe23c3cf
I20240503 02:11:45.711760 26705 raft_consensus.cc:1203] T 7b671abb263d4848b7b909927b7df380 P 03afb03fe005462eb07ada37fe23c3cf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:11:45.713793 26860 raft_consensus.cc:2932] T 7b671abb263d4848b7b909927b7df380 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 03afb03fe005462eb07ada37fe23c3cf (127.22.108.7) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "03afb03fe005462eb07ada37fe23c3cf" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 34851 } } }
I20240503 02:11:45.714447 26705 raft_consensus.cc:2932] T 7b671abb263d4848b7b909927b7df380 P 03afb03fe005462eb07ada37fe23c3cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 03afb03fe005462eb07ada37fe23c3cf (127.22.108.7) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "03afb03fe005462eb07ada37fe23c3cf" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 34851 } } }
I20240503 02:11:45.717489 26860 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:48684 (request call id 0) took 1031 ms. Trace:
I20240503 02:11:45.717705 26860 rpcz_store.cc:276] 0503 02:11:44.685647 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:11:44.685714 (+ 67us) service_pool.cc:225] Handling call
0503 02:11:44.687738 (+ 2024us) log.cc:842] Serialized 257 byte log entry
0503 02:11:45.717441 (+1029703us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":62,"raft.queue_time_us":362,"raft.run_cpu_time_us":465,"raft.run_wall_time_us":465,"spinlock_wait_cycles":55552,"thread_start_us":433,"threads_started":2,"wal-append.queue_time_us":269,"wal-append.run_cpu_time_us":547,"wal-append.run_wall_time_us":1000730}
I20240503 02:11:45.720502 24846 catalog_manager.cc:5484] T 7b671abb263d4848b7b909927b7df380 P 22b0b1ae6c504301ad5170c6bdcd179e reported cstate change: config changed from index -1 to 4, VOTER 03afb03fe005462eb07ada37fe23c3cf (127.22.108.7) added. New cstate: current_term: 1 leader_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "03afb03fe005462eb07ada37fe23c3cf" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 34851 } } }
I20240503 02:11:45.753435 24990 consensus_queue.cc:237] T 7c0b16db9b77442b922e3b7bf1247878 P 308436a6f7c7456b99971c666d3bd049 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "0163beb78bfd4d399a70ebacb8f9aa9c" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 45401 } }
W20240503 02:11:45.762849 24924 consensus_peers.cc:485] T 7c0b16db9b77442b922e3b7bf1247878 P 308436a6f7c7456b99971c666d3bd049 -> Peer 0163beb78bfd4d399a70ebacb8f9aa9c (127.22.108.4:45401): Couldn't send request to peer 0163beb78bfd4d399a70ebacb8f9aa9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7c0b16db9b77442b922e3b7bf1247878. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:46.169503 26888 ts_tablet_manager.cc:902] T 7c0b16db9b77442b922e3b7bf1247878 P 0163beb78bfd4d399a70ebacb8f9aa9c: Initiating tablet copy from peer 308436a6f7c7456b99971c666d3bd049 (127.22.108.1:40941)
I20240503 02:11:46.171280 26888 tablet_copy_client.cc:308] T 7c0b16db9b77442b922e3b7bf1247878 P 0163beb78bfd4d399a70ebacb8f9aa9c: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.1:40941
I20240503 02:11:46.178602 25010 tablet_copy_service.cc:140] P 308436a6f7c7456b99971c666d3bd049: Received BeginTabletCopySession request for tablet 7c0b16db9b77442b922e3b7bf1247878 from peer 0163beb78bfd4d399a70ebacb8f9aa9c ({username='slave'} at 127.22.108.4:40093)
I20240503 02:11:46.178870 25010 tablet_copy_service.cc:161] P 308436a6f7c7456b99971c666d3bd049: Beginning new tablet copy session on tablet 7c0b16db9b77442b922e3b7bf1247878 from peer 0163beb78bfd4d399a70ebacb8f9aa9c at {username='slave'} at 127.22.108.4:40093: session id = 0163beb78bfd4d399a70ebacb8f9aa9c-7c0b16db9b77442b922e3b7bf1247878
I20240503 02:11:46.181674 25010 tablet_copy_source_session.cc:205] T 7c0b16db9b77442b922e3b7bf1247878 P 308436a6f7c7456b99971c666d3bd049: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:11:46.186235 26888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c0b16db9b77442b922e3b7bf1247878. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:46.196632 26888 tablet_copy_client.cc:783] T 7c0b16db9b77442b922e3b7bf1247878 P 0163beb78bfd4d399a70ebacb8f9aa9c: tablet copy: Starting download of 0 data blocks...
I20240503 02:11:46.196959 26888 tablet_copy_client.cc:647] T 7c0b16db9b77442b922e3b7bf1247878 P 0163beb78bfd4d399a70ebacb8f9aa9c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:11:46.199290 26888 tablet_copy_client.cc:523] T 7c0b16db9b77442b922e3b7bf1247878 P 0163beb78bfd4d399a70ebacb8f9aa9c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:11:46.202239 26888 tablet_bootstrap.cc:492] T 7c0b16db9b77442b922e3b7bf1247878 P 0163beb78bfd4d399a70ebacb8f9aa9c: Bootstrap starting.
I20240503 02:11:46.211293 26888 log.cc:824] T 7c0b16db9b77442b922e3b7bf1247878 P 0163beb78bfd4d399a70ebacb8f9aa9c: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:46.250910 26888 tablet_bootstrap.cc:492] T 7c0b16db9b77442b922e3b7bf1247878 P 0163beb78bfd4d399a70ebacb8f9aa9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 02:11:46.251446 26888 tablet_bootstrap.cc:492] T 7c0b16db9b77442b922e3b7bf1247878 P 0163beb78bfd4d399a70ebacb8f9aa9c: Bootstrap complete.
I20240503 02:11:46.251837 26888 ts_tablet_manager.cc:1367] T 7c0b16db9b77442b922e3b7bf1247878 P 0163beb78bfd4d399a70ebacb8f9aa9c: Time spent bootstrapping tablet: real 0.050s user 0.042s sys 0.006s
I20240503 02:11:46.256302 26888 raft_consensus.cc:348] T 7c0b16db9b77442b922e3b7bf1247878 P 0163beb78bfd4d399a70ebacb8f9aa9c [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:46.257179 26888 raft_consensus.cc:729] T 7c0b16db9b77442b922e3b7bf1247878 P 0163beb78bfd4d399a70ebacb8f9aa9c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0163beb78bfd4d399a70ebacb8f9aa9c, State: Initialized, Role: FOLLOWER
I20240503 02:11:46.257608 26888 consensus_queue.cc:260] T 7c0b16db9b77442b922e3b7bf1247878 P 0163beb78bfd4d399a70ebacb8f9aa9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "0163beb78bfd4d399a70ebacb8f9aa9c" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 45401 } }
I20240503 02:11:46.259138 26888 ts_tablet_manager.cc:1398] T 7c0b16db9b77442b922e3b7bf1247878 P 0163beb78bfd4d399a70ebacb8f9aa9c: Time spent starting tablet: real 0.007s user 0.005s sys 0.004s
I20240503 02:11:46.260217 25010 tablet_copy_service.cc:342] P 308436a6f7c7456b99971c666d3bd049: Request end of tablet copy session 0163beb78bfd4d399a70ebacb8f9aa9c-7c0b16db9b77442b922e3b7bf1247878 received from {username='slave'} at 127.22.108.4:40093
I20240503 02:11:46.260389 25010 tablet_copy_service.cc:433] P 308436a6f7c7456b99971c666d3bd049: ending tablet copy session 0163beb78bfd4d399a70ebacb8f9aa9c-7c0b16db9b77442b922e3b7bf1247878 on tablet 7c0b16db9b77442b922e3b7bf1247878 with peer 0163beb78bfd4d399a70ebacb8f9aa9c
I20240503 02:11:46.560134 26315 raft_consensus.cc:1203] T 7c0b16db9b77442b922e3b7bf1247878 P 0163beb78bfd4d399a70ebacb8f9aa9c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:11:46.561848 26897 raft_consensus.cc:2932] T 7c0b16db9b77442b922e3b7bf1247878 P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0163beb78bfd4d399a70ebacb8f9aa9c (127.22.108.4) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "0163beb78bfd4d399a70ebacb8f9aa9c" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 45401 } } }
I20240503 02:11:46.562721 26315 raft_consensus.cc:2932] T 7c0b16db9b77442b922e3b7bf1247878 P 0163beb78bfd4d399a70ebacb8f9aa9c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0163beb78bfd4d399a70ebacb8f9aa9c (127.22.108.4) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "0163beb78bfd4d399a70ebacb8f9aa9c" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 45401 } } }
I20240503 02:11:46.568583 24846 catalog_manager.cc:5484] T 7c0b16db9b77442b922e3b7bf1247878 P 0163beb78bfd4d399a70ebacb8f9aa9c reported cstate change: config changed from index -1 to 4, VOTER 0163beb78bfd4d399a70ebacb8f9aa9c (127.22.108.4) added. New cstate: current_term: 1 leader_uuid: "308436a6f7c7456b99971c666d3bd049" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "0163beb78bfd4d399a70ebacb8f9aa9c" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 45401 } } }
I20240503 02:11:46.605947 25121 consensus_queue.cc:237] T bb1a9efdfd96421ea17c41cfa8977f36 P 22b0b1ae6c504301ad5170c6bdcd179e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } }
W20240503 02:11:46.614003 25056 consensus_peers.cc:485] T bb1a9efdfd96421ea17c41cfa8977f36 P 22b0b1ae6c504301ad5170c6bdcd179e -> Peer 6c704dfc19814db996a1c0e363e760ef (127.22.108.6:43731): Couldn't send request to peer 6c704dfc19814db996a1c0e363e760ef. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb1a9efdfd96421ea17c41cfa8977f36. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:47.085811 26922 ts_tablet_manager.cc:902] T bb1a9efdfd96421ea17c41cfa8977f36 P 6c704dfc19814db996a1c0e363e760ef: Initiating tablet copy from peer 22b0b1ae6c504301ad5170c6bdcd179e (127.22.108.2:36213)
I20240503 02:11:47.087641 26922 tablet_copy_client.cc:308] T bb1a9efdfd96421ea17c41cfa8977f36 P 6c704dfc19814db996a1c0e363e760ef: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.2:36213
I20240503 02:11:47.094280 25141 tablet_copy_service.cc:140] P 22b0b1ae6c504301ad5170c6bdcd179e: Received BeginTabletCopySession request for tablet bb1a9efdfd96421ea17c41cfa8977f36 from peer 6c704dfc19814db996a1c0e363e760ef ({username='slave'} at 127.22.108.6:45763)
I20240503 02:11:47.094480 25141 tablet_copy_service.cc:161] P 22b0b1ae6c504301ad5170c6bdcd179e: Beginning new tablet copy session on tablet bb1a9efdfd96421ea17c41cfa8977f36 from peer 6c704dfc19814db996a1c0e363e760ef at {username='slave'} at 127.22.108.6:45763: session id = 6c704dfc19814db996a1c0e363e760ef-bb1a9efdfd96421ea17c41cfa8977f36
I20240503 02:11:47.096333 25141 tablet_copy_source_session.cc:205] T bb1a9efdfd96421ea17c41cfa8977f36 P 22b0b1ae6c504301ad5170c6bdcd179e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:11:47.100812 26922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb1a9efdfd96421ea17c41cfa8977f36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:47.111465 26922 tablet_copy_client.cc:783] T bb1a9efdfd96421ea17c41cfa8977f36 P 6c704dfc19814db996a1c0e363e760ef: tablet copy: Starting download of 0 data blocks...
I20240503 02:11:47.111856 26922 tablet_copy_client.cc:647] T bb1a9efdfd96421ea17c41cfa8977f36 P 6c704dfc19814db996a1c0e363e760ef: tablet copy: Starting download of 1 WAL segments...
I20240503 02:11:47.114102 26922 tablet_copy_client.cc:523] T bb1a9efdfd96421ea17c41cfa8977f36 P 6c704dfc19814db996a1c0e363e760ef: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:11:47.117164 26922 tablet_bootstrap.cc:492] T bb1a9efdfd96421ea17c41cfa8977f36 P 6c704dfc19814db996a1c0e363e760ef: Bootstrap starting.
I20240503 02:11:47.126888 26922 log.cc:824] T bb1a9efdfd96421ea17c41cfa8977f36 P 6c704dfc19814db996a1c0e363e760ef: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:47.158607 26922 tablet_bootstrap.cc:492] T bb1a9efdfd96421ea17c41cfa8977f36 P 6c704dfc19814db996a1c0e363e760ef: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 02:11:47.159191 26922 tablet_bootstrap.cc:492] T bb1a9efdfd96421ea17c41cfa8977f36 P 6c704dfc19814db996a1c0e363e760ef: Bootstrap complete.
I20240503 02:11:47.159654 26922 ts_tablet_manager.cc:1367] T bb1a9efdfd96421ea17c41cfa8977f36 P 6c704dfc19814db996a1c0e363e760ef: Time spent bootstrapping tablet: real 0.043s user 0.038s sys 0.004s
I20240503 02:11:47.164198 26922 raft_consensus.cc:348] T bb1a9efdfd96421ea17c41cfa8977f36 P 6c704dfc19814db996a1c0e363e760ef [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } }
I20240503 02:11:47.165139 26922 raft_consensus.cc:729] T bb1a9efdfd96421ea17c41cfa8977f36 P 6c704dfc19814db996a1c0e363e760ef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c704dfc19814db996a1c0e363e760ef, State: Initialized, Role: FOLLOWER
I20240503 02:11:47.165603 26922 consensus_queue.cc:260] T bb1a9efdfd96421ea17c41cfa8977f36 P 6c704dfc19814db996a1c0e363e760ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } }
I20240503 02:11:47.167271 26922 ts_tablet_manager.cc:1398] T bb1a9efdfd96421ea17c41cfa8977f36 P 6c704dfc19814db996a1c0e363e760ef: Time spent starting tablet: real 0.008s user 0.006s sys 0.003s
I20240503 02:11:47.168406 25141 tablet_copy_service.cc:342] P 22b0b1ae6c504301ad5170c6bdcd179e: Request end of tablet copy session 6c704dfc19814db996a1c0e363e760ef-bb1a9efdfd96421ea17c41cfa8977f36 received from {username='slave'} at 127.22.108.6:45763
I20240503 02:11:47.168534 25141 tablet_copy_service.cc:433] P 22b0b1ae6c504301ad5170c6bdcd179e: ending tablet copy session 6c704dfc19814db996a1c0e363e760ef-bb1a9efdfd96421ea17c41cfa8977f36 on tablet bb1a9efdfd96421ea17c41cfa8977f36 with peer 6c704dfc19814db996a1c0e363e760ef
I20240503 02:11:47.523594 26575 raft_consensus.cc:1203] T bb1a9efdfd96421ea17c41cfa8977f36 P 6c704dfc19814db996a1c0e363e760ef [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:11:47.525483 26931 raft_consensus.cc:2932] T bb1a9efdfd96421ea17c41cfa8977f36 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6c704dfc19814db996a1c0e363e760ef (127.22.108.6) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } } }
I20240503 02:11:47.526189 26575 raft_consensus.cc:2932] T bb1a9efdfd96421ea17c41cfa8977f36 P 6c704dfc19814db996a1c0e363e760ef [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6c704dfc19814db996a1c0e363e760ef (127.22.108.6) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } } }
I20240503 02:11:47.531250 24848 catalog_manager.cc:5484] T bb1a9efdfd96421ea17c41cfa8977f36 P 22b0b1ae6c504301ad5170c6bdcd179e reported cstate change: config changed from index -1 to 4, VOTER 6c704dfc19814db996a1c0e363e760ef (127.22.108.6) added. New cstate: current_term: 1 leader_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } } }
I20240503 02:11:47.560318 24990 consensus_queue.cc:237] T 923672905ffb452993c5dcf283f4a399 P 308436a6f7c7456b99971c666d3bd049 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "1b2a71b48d4e40679504230af08279e9" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 40741 } }
W20240503 02:11:47.568408 24923 consensus_peers.cc:485] T 923672905ffb452993c5dcf283f4a399 P 308436a6f7c7456b99971c666d3bd049 -> Peer 1b2a71b48d4e40679504230af08279e9 (127.22.108.5:40741): Couldn't send request to peer 1b2a71b48d4e40679504230af08279e9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 923672905ffb452993c5dcf283f4a399. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:48.127624 26956 ts_tablet_manager.cc:902] T 923672905ffb452993c5dcf283f4a399 P 1b2a71b48d4e40679504230af08279e9: Initiating tablet copy from peer 308436a6f7c7456b99971c666d3bd049 (127.22.108.1:40941)
I20240503 02:11:48.129484 26956 tablet_copy_client.cc:308] T 923672905ffb452993c5dcf283f4a399 P 1b2a71b48d4e40679504230af08279e9: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.1:40941
I20240503 02:11:48.136482 25010 tablet_copy_service.cc:140] P 308436a6f7c7456b99971c666d3bd049: Received BeginTabletCopySession request for tablet 923672905ffb452993c5dcf283f4a399 from peer 1b2a71b48d4e40679504230af08279e9 ({username='slave'} at 127.22.108.5:40821)
I20240503 02:11:48.136672 25010 tablet_copy_service.cc:161] P 308436a6f7c7456b99971c666d3bd049: Beginning new tablet copy session on tablet 923672905ffb452993c5dcf283f4a399 from peer 1b2a71b48d4e40679504230af08279e9 at {username='slave'} at 127.22.108.5:40821: session id = 1b2a71b48d4e40679504230af08279e9-923672905ffb452993c5dcf283f4a399
I20240503 02:11:48.138670 25010 tablet_copy_source_session.cc:205] T 923672905ffb452993c5dcf283f4a399 P 308436a6f7c7456b99971c666d3bd049: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:11:48.142963 26956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 923672905ffb452993c5dcf283f4a399. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:48.153136 26956 tablet_copy_client.cc:783] T 923672905ffb452993c5dcf283f4a399 P 1b2a71b48d4e40679504230af08279e9: tablet copy: Starting download of 0 data blocks...
I20240503 02:11:48.153460 26956 tablet_copy_client.cc:647] T 923672905ffb452993c5dcf283f4a399 P 1b2a71b48d4e40679504230af08279e9: tablet copy: Starting download of 1 WAL segments...
I20240503 02:11:48.155570 26956 tablet_copy_client.cc:523] T 923672905ffb452993c5dcf283f4a399 P 1b2a71b48d4e40679504230af08279e9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:11:48.158725 26956 tablet_bootstrap.cc:492] T 923672905ffb452993c5dcf283f4a399 P 1b2a71b48d4e40679504230af08279e9: Bootstrap starting.
I20240503 02:11:48.168401 26956 log.cc:824] T 923672905ffb452993c5dcf283f4a399 P 1b2a71b48d4e40679504230af08279e9: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:48.199165 26956 tablet_bootstrap.cc:492] T 923672905ffb452993c5dcf283f4a399 P 1b2a71b48d4e40679504230af08279e9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 02:11:48.199752 26956 tablet_bootstrap.cc:492] T 923672905ffb452993c5dcf283f4a399 P 1b2a71b48d4e40679504230af08279e9: Bootstrap complete.
I20240503 02:11:48.200187 26956 ts_tablet_manager.cc:1367] T 923672905ffb452993c5dcf283f4a399 P 1b2a71b48d4e40679504230af08279e9: Time spent bootstrapping tablet: real 0.042s user 0.031s sys 0.009s
I20240503 02:11:48.204906 26956 raft_consensus.cc:348] T 923672905ffb452993c5dcf283f4a399 P 1b2a71b48d4e40679504230af08279e9 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:48.205742 26956 raft_consensus.cc:729] T 923672905ffb452993c5dcf283f4a399 P 1b2a71b48d4e40679504230af08279e9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b2a71b48d4e40679504230af08279e9, State: Initialized, Role: FOLLOWER
I20240503 02:11:48.206132 26956 consensus_queue.cc:260] T 923672905ffb452993c5dcf283f4a399 P 1b2a71b48d4e40679504230af08279e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "1b2a71b48d4e40679504230af08279e9" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 40741 } }
I20240503 02:11:48.207597 26956 ts_tablet_manager.cc:1398] T 923672905ffb452993c5dcf283f4a399 P 1b2a71b48d4e40679504230af08279e9: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
I20240503 02:11:48.208745 25010 tablet_copy_service.cc:342] P 308436a6f7c7456b99971c666d3bd049: Request end of tablet copy session 1b2a71b48d4e40679504230af08279e9-923672905ffb452993c5dcf283f4a399 received from {username='slave'} at 127.22.108.5:40821
I20240503 02:11:48.208873 25010 tablet_copy_service.cc:433] P 308436a6f7c7456b99971c666d3bd049: ending tablet copy session 1b2a71b48d4e40679504230af08279e9-923672905ffb452993c5dcf283f4a399 on tablet 923672905ffb452993c5dcf283f4a399 with peer 1b2a71b48d4e40679504230af08279e9
I20240503 02:11:48.549953 26445 raft_consensus.cc:1203] T 923672905ffb452993c5dcf283f4a399 P 1b2a71b48d4e40679504230af08279e9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:11:48.551654 26966 raft_consensus.cc:2932] T 923672905ffb452993c5dcf283f4a399 P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1b2a71b48d4e40679504230af08279e9 (127.22.108.5) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "1b2a71b48d4e40679504230af08279e9" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 40741 } } }
I20240503 02:11:48.552489 26445 raft_consensus.cc:2932] T 923672905ffb452993c5dcf283f4a399 P 1b2a71b48d4e40679504230af08279e9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1b2a71b48d4e40679504230af08279e9 (127.22.108.5) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "1b2a71b48d4e40679504230af08279e9" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 40741 } } }
I20240503 02:11:48.556807 24846 catalog_manager.cc:5484] T 923672905ffb452993c5dcf283f4a399 P 308436a6f7c7456b99971c666d3bd049 reported cstate change: config changed from index -1 to 4, VOTER 1b2a71b48d4e40679504230af08279e9 (127.22.108.5) added. New cstate: current_term: 1 leader_uuid: "308436a6f7c7456b99971c666d3bd049" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "1b2a71b48d4e40679504230af08279e9" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 40741 } } }
I20240503 02:11:48.594300 25121 consensus_queue.cc:237] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "b95ea99cb72f49c6b42c7e8e293cfc27" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 36823 } }
W20240503 02:11:48.602044 25054 consensus_peers.cc:485] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e -> Peer b95ea99cb72f49c6b42c7e8e293cfc27 (127.22.108.3:36823): Couldn't send request to peer b95ea99cb72f49c6b42c7e8e293cfc27. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ee1a68079bfa435ab820967fb894d3f1. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:49.129341 26992 ts_tablet_manager.cc:902] T ee1a68079bfa435ab820967fb894d3f1 P b95ea99cb72f49c6b42c7e8e293cfc27: Initiating tablet copy from peer 22b0b1ae6c504301ad5170c6bdcd179e (127.22.108.2:36213)
I20240503 02:11:49.130908 26992 tablet_copy_client.cc:308] T ee1a68079bfa435ab820967fb894d3f1 P b95ea99cb72f49c6b42c7e8e293cfc27: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.2:36213
I20240503 02:11:49.137378 25141 tablet_copy_service.cc:140] P 22b0b1ae6c504301ad5170c6bdcd179e: Received BeginTabletCopySession request for tablet ee1a68079bfa435ab820967fb894d3f1 from peer b95ea99cb72f49c6b42c7e8e293cfc27 ({username='slave'} at 127.22.108.3:58211)
I20240503 02:11:49.137553 25141 tablet_copy_service.cc:161] P 22b0b1ae6c504301ad5170c6bdcd179e: Beginning new tablet copy session on tablet ee1a68079bfa435ab820967fb894d3f1 from peer b95ea99cb72f49c6b42c7e8e293cfc27 at {username='slave'} at 127.22.108.3:58211: session id = b95ea99cb72f49c6b42c7e8e293cfc27-ee1a68079bfa435ab820967fb894d3f1
I20240503 02:11:49.139441 25141 tablet_copy_source_session.cc:205] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:11:49.143299 26992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee1a68079bfa435ab820967fb894d3f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:49.153952 26992 tablet_copy_client.cc:783] T ee1a68079bfa435ab820967fb894d3f1 P b95ea99cb72f49c6b42c7e8e293cfc27: tablet copy: Starting download of 0 data blocks...
I20240503 02:11:49.154258 26992 tablet_copy_client.cc:647] T ee1a68079bfa435ab820967fb894d3f1 P b95ea99cb72f49c6b42c7e8e293cfc27: tablet copy: Starting download of 1 WAL segments...
I20240503 02:11:49.156134 26992 tablet_copy_client.cc:523] T ee1a68079bfa435ab820967fb894d3f1 P b95ea99cb72f49c6b42c7e8e293cfc27: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:11:49.159070 26992 tablet_bootstrap.cc:492] T ee1a68079bfa435ab820967fb894d3f1 P b95ea99cb72f49c6b42c7e8e293cfc27: Bootstrap starting.
I20240503 02:11:49.165308 26992 log.cc:824] T ee1a68079bfa435ab820967fb894d3f1 P b95ea99cb72f49c6b42c7e8e293cfc27: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:49.168529 26992 tablet_bootstrap.cc:492] T ee1a68079bfa435ab820967fb894d3f1 P b95ea99cb72f49c6b42c7e8e293cfc27: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 02:11:49.168957 26992 tablet_bootstrap.cc:492] T ee1a68079bfa435ab820967fb894d3f1 P b95ea99cb72f49c6b42c7e8e293cfc27: Bootstrap complete.
I20240503 02:11:49.169328 26992 ts_tablet_manager.cc:1367] T ee1a68079bfa435ab820967fb894d3f1 P b95ea99cb72f49c6b42c7e8e293cfc27: Time spent bootstrapping tablet: real 0.010s user 0.011s sys 0.000s
I20240503 02:11:49.175530 26992 raft_consensus.cc:348] T ee1a68079bfa435ab820967fb894d3f1 P b95ea99cb72f49c6b42c7e8e293cfc27 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } }
I20240503 02:11:49.176322 26992 raft_consensus.cc:729] T ee1a68079bfa435ab820967fb894d3f1 P b95ea99cb72f49c6b42c7e8e293cfc27 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95ea99cb72f49c6b42c7e8e293cfc27, State: Initialized, Role: FOLLOWER
I20240503 02:11:49.176702 26992 consensus_queue.cc:260] T ee1a68079bfa435ab820967fb894d3f1 P b95ea99cb72f49c6b42c7e8e293cfc27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "b95ea99cb72f49c6b42c7e8e293cfc27" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 36823 } }
I20240503 02:11:49.178014 26992 ts_tablet_manager.cc:1398] T ee1a68079bfa435ab820967fb894d3f1 P b95ea99cb72f49c6b42c7e8e293cfc27: Time spent starting tablet: real 0.009s user 0.005s sys 0.003s
I20240503 02:11:49.178910 25141 tablet_copy_service.cc:342] P 22b0b1ae6c504301ad5170c6bdcd179e: Request end of tablet copy session b95ea99cb72f49c6b42c7e8e293cfc27-ee1a68079bfa435ab820967fb894d3f1 received from {username='slave'} at 127.22.108.3:58211
I20240503 02:11:49.179049 25141 tablet_copy_service.cc:433] P 22b0b1ae6c504301ad5170c6bdcd179e: ending tablet copy session b95ea99cb72f49c6b42c7e8e293cfc27-ee1a68079bfa435ab820967fb894d3f1 on tablet ee1a68079bfa435ab820967fb894d3f1 with peer b95ea99cb72f49c6b42c7e8e293cfc27
I20240503 02:11:49.506955 26184 raft_consensus.cc:1203] T ee1a68079bfa435ab820967fb894d3f1 P b95ea99cb72f49c6b42c7e8e293cfc27 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:11:49.508826 27002 raft_consensus.cc:2932] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b95ea99cb72f49c6b42c7e8e293cfc27 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "b95ea99cb72f49c6b42c7e8e293cfc27" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 36823 } } }
I20240503 02:11:49.509721 26184 raft_consensus.cc:2932] T ee1a68079bfa435ab820967fb894d3f1 P b95ea99cb72f49c6b42c7e8e293cfc27 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b95ea99cb72f49c6b42c7e8e293cfc27 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "b95ea99cb72f49c6b42c7e8e293cfc27" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 36823 } } }
I20240503 02:11:49.515434 24846 catalog_manager.cc:5484] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e reported cstate change: config changed from index -1 to 2, VOTER b95ea99cb72f49c6b42c7e8e293cfc27 (127.22.108.3) added. New cstate: current_term: 1 leader_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "b95ea99cb72f49c6b42c7e8e293cfc27" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 36823 } } }
I20240503 02:11:49.553339 24990 consensus_queue.cc:237] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 308436a6f7c7456b99971c666d3bd049 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "03afb03fe005462eb07ada37fe23c3cf" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 34851 } }
W20240503 02:11:49.560550 24924 consensus_peers.cc:485] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 308436a6f7c7456b99971c666d3bd049 -> Peer 03afb03fe005462eb07ada37fe23c3cf (127.22.108.7:34851): Couldn't send request to peer 03afb03fe005462eb07ada37fe23c3cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ed43f867d21d4bee9efc0b9bf4ee90c1. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:49.970276 27027 ts_tablet_manager.cc:902] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 03afb03fe005462eb07ada37fe23c3cf: Initiating tablet copy from peer 308436a6f7c7456b99971c666d3bd049 (127.22.108.1:40941)
I20240503 02:11:49.971402 27027 tablet_copy_client.cc:308] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 03afb03fe005462eb07ada37fe23c3cf: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.1:40941
I20240503 02:11:49.977134 25010 tablet_copy_service.cc:140] P 308436a6f7c7456b99971c666d3bd049: Received BeginTabletCopySession request for tablet ed43f867d21d4bee9efc0b9bf4ee90c1 from peer 03afb03fe005462eb07ada37fe23c3cf ({username='slave'} at 127.22.108.7:43543)
I20240503 02:11:49.977274 25010 tablet_copy_service.cc:161] P 308436a6f7c7456b99971c666d3bd049: Beginning new tablet copy session on tablet ed43f867d21d4bee9efc0b9bf4ee90c1 from peer 03afb03fe005462eb07ada37fe23c3cf at {username='slave'} at 127.22.108.7:43543: session id = 03afb03fe005462eb07ada37fe23c3cf-ed43f867d21d4bee9efc0b9bf4ee90c1
I20240503 02:11:49.979116 25010 tablet_copy_source_session.cc:205] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 308436a6f7c7456b99971c666d3bd049: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:11:49.980232 27027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed43f867d21d4bee9efc0b9bf4ee90c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:49.985159 27027 tablet_copy_client.cc:783] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 03afb03fe005462eb07ada37fe23c3cf: tablet copy: Starting download of 0 data blocks...
I20240503 02:11:49.985460 27027 tablet_copy_client.cc:647] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 03afb03fe005462eb07ada37fe23c3cf: tablet copy: Starting download of 1 WAL segments...
I20240503 02:11:49.987519 27027 tablet_copy_client.cc:523] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 03afb03fe005462eb07ada37fe23c3cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:11:49.990767 27027 tablet_bootstrap.cc:492] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 03afb03fe005462eb07ada37fe23c3cf: Bootstrap starting.
I20240503 02:11:50.019337 27027 tablet_bootstrap.cc:492] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 03afb03fe005462eb07ada37fe23c3cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 02:11:50.019711 27027 tablet_bootstrap.cc:492] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 03afb03fe005462eb07ada37fe23c3cf: Bootstrap complete.
I20240503 02:11:50.019920 27027 ts_tablet_manager.cc:1367] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 03afb03fe005462eb07ada37fe23c3cf: Time spent bootstrapping tablet: real 0.029s user 0.020s sys 0.007s
I20240503 02:11:50.020682 27027 raft_consensus.cc:348] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 03afb03fe005462eb07ada37fe23c3cf [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:50.020990 27027 raft_consensus.cc:729] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 03afb03fe005462eb07ada37fe23c3cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03afb03fe005462eb07ada37fe23c3cf, State: Initialized, Role: FOLLOWER
I20240503 02:11:50.021173 27027 consensus_queue.cc:260] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 03afb03fe005462eb07ada37fe23c3cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "03afb03fe005462eb07ada37fe23c3cf" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 34851 } }
I20240503 02:11:50.021850 27027 ts_tablet_manager.cc:1398] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 03afb03fe005462eb07ada37fe23c3cf: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 02:11:50.022797 25010 tablet_copy_service.cc:342] P 308436a6f7c7456b99971c666d3bd049: Request end of tablet copy session 03afb03fe005462eb07ada37fe23c3cf-ed43f867d21d4bee9efc0b9bf4ee90c1 received from {username='slave'} at 127.22.108.7:43543
I20240503 02:11:50.022936 25010 tablet_copy_service.cc:433] P 308436a6f7c7456b99971c666d3bd049: ending tablet copy session 03afb03fe005462eb07ada37fe23c3cf-ed43f867d21d4bee9efc0b9bf4ee90c1 on tablet ed43f867d21d4bee9efc0b9bf4ee90c1 with peer 03afb03fe005462eb07ada37fe23c3cf
I20240503 02:11:50.538702 26705 raft_consensus.cc:1203] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 03afb03fe005462eb07ada37fe23c3cf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:11:50.540433 27036 raft_consensus.cc:2932] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 03afb03fe005462eb07ada37fe23c3cf (127.22.108.7) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "03afb03fe005462eb07ada37fe23c3cf" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 34851 } } }
I20240503 02:11:50.543422 26705 raft_consensus.cc:2932] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 03afb03fe005462eb07ada37fe23c3cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 03afb03fe005462eb07ada37fe23c3cf (127.22.108.7) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "03afb03fe005462eb07ada37fe23c3cf" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 34851 } } }
I20240503 02:11:50.549813 24848 catalog_manager.cc:5484] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 308436a6f7c7456b99971c666d3bd049 reported cstate change: config changed from index -1 to 4, VOTER 03afb03fe005462eb07ada37fe23c3cf (127.22.108.7) added. New cstate: current_term: 1 leader_uuid: "308436a6f7c7456b99971c666d3bd049" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "03afb03fe005462eb07ada37fe23c3cf" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 34851 } } }
I20240503 02:11:50.599129 24990 consensus_queue.cc:237] T f6920bfd97624ad381fd72b688ab73f1 P 308436a6f7c7456b99971c666d3bd049 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "0163beb78bfd4d399a70ebacb8f9aa9c" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 45401 } }
W20240503 02:11:50.601107 24924 consensus_peers.cc:485] T f6920bfd97624ad381fd72b688ab73f1 P 308436a6f7c7456b99971c666d3bd049 -> Peer 0163beb78bfd4d399a70ebacb8f9aa9c (127.22.108.4:45401): Couldn't send request to peer 0163beb78bfd4d399a70ebacb8f9aa9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f6920bfd97624ad381fd72b688ab73f1. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:51.043928 27060 ts_tablet_manager.cc:902] T f6920bfd97624ad381fd72b688ab73f1 P 0163beb78bfd4d399a70ebacb8f9aa9c: Initiating tablet copy from peer 308436a6f7c7456b99971c666d3bd049 (127.22.108.1:40941)
I20240503 02:11:51.045125 27060 tablet_copy_client.cc:308] T f6920bfd97624ad381fd72b688ab73f1 P 0163beb78bfd4d399a70ebacb8f9aa9c: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.1:40941
I20240503 02:11:51.045830 25010 tablet_copy_service.cc:140] P 308436a6f7c7456b99971c666d3bd049: Received BeginTabletCopySession request for tablet f6920bfd97624ad381fd72b688ab73f1 from peer 0163beb78bfd4d399a70ebacb8f9aa9c ({username='slave'} at 127.22.108.4:40093)
I20240503 02:11:51.046005 25010 tablet_copy_service.cc:161] P 308436a6f7c7456b99971c666d3bd049: Beginning new tablet copy session on tablet f6920bfd97624ad381fd72b688ab73f1 from peer 0163beb78bfd4d399a70ebacb8f9aa9c at {username='slave'} at 127.22.108.4:40093: session id = 0163beb78bfd4d399a70ebacb8f9aa9c-f6920bfd97624ad381fd72b688ab73f1
I20240503 02:11:51.048406 25010 tablet_copy_source_session.cc:205] T f6920bfd97624ad381fd72b688ab73f1 P 308436a6f7c7456b99971c666d3bd049: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:11:51.049641 27060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6920bfd97624ad381fd72b688ab73f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:51.054080 27060 tablet_copy_client.cc:783] T f6920bfd97624ad381fd72b688ab73f1 P 0163beb78bfd4d399a70ebacb8f9aa9c: tablet copy: Starting download of 0 data blocks...
I20240503 02:11:51.054313 27060 tablet_copy_client.cc:647] T f6920bfd97624ad381fd72b688ab73f1 P 0163beb78bfd4d399a70ebacb8f9aa9c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:11:51.055881 27060 tablet_copy_client.cc:523] T f6920bfd97624ad381fd72b688ab73f1 P 0163beb78bfd4d399a70ebacb8f9aa9c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:11:51.058161 27060 tablet_bootstrap.cc:492] T f6920bfd97624ad381fd72b688ab73f1 P 0163beb78bfd4d399a70ebacb8f9aa9c: Bootstrap starting.
I20240503 02:11:51.095052 27060 tablet_bootstrap.cc:492] T f6920bfd97624ad381fd72b688ab73f1 P 0163beb78bfd4d399a70ebacb8f9aa9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 02:11:51.095466 27060 tablet_bootstrap.cc:492] T f6920bfd97624ad381fd72b688ab73f1 P 0163beb78bfd4d399a70ebacb8f9aa9c: Bootstrap complete.
I20240503 02:11:51.096201 27060 ts_tablet_manager.cc:1367] T f6920bfd97624ad381fd72b688ab73f1 P 0163beb78bfd4d399a70ebacb8f9aa9c: Time spent bootstrapping tablet: real 0.038s user 0.027s sys 0.008s
I20240503 02:11:51.096983 27060 raft_consensus.cc:348] T f6920bfd97624ad381fd72b688ab73f1 P 0163beb78bfd4d399a70ebacb8f9aa9c [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:51.097355 27060 raft_consensus.cc:729] T f6920bfd97624ad381fd72b688ab73f1 P 0163beb78bfd4d399a70ebacb8f9aa9c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0163beb78bfd4d399a70ebacb8f9aa9c, State: Initialized, Role: FOLLOWER
I20240503 02:11:51.097537 27060 consensus_queue.cc:260] T f6920bfd97624ad381fd72b688ab73f1 P 0163beb78bfd4d399a70ebacb8f9aa9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "0163beb78bfd4d399a70ebacb8f9aa9c" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 45401 } }
I20240503 02:11:51.098181 27060 ts_tablet_manager.cc:1398] T f6920bfd97624ad381fd72b688ab73f1 P 0163beb78bfd4d399a70ebacb8f9aa9c: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240503 02:11:51.099047 25010 tablet_copy_service.cc:342] P 308436a6f7c7456b99971c666d3bd049: Request end of tablet copy session 0163beb78bfd4d399a70ebacb8f9aa9c-f6920bfd97624ad381fd72b688ab73f1 received from {username='slave'} at 127.22.108.4:40093
I20240503 02:11:51.099155 25010 tablet_copy_service.cc:433] P 308436a6f7c7456b99971c666d3bd049: ending tablet copy session 0163beb78bfd4d399a70ebacb8f9aa9c-f6920bfd97624ad381fd72b688ab73f1 on tablet f6920bfd97624ad381fd72b688ab73f1 with peer 0163beb78bfd4d399a70ebacb8f9aa9c
I20240503 02:11:51.657146 26315 raft_consensus.cc:1203] T f6920bfd97624ad381fd72b688ab73f1 P 0163beb78bfd4d399a70ebacb8f9aa9c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:11:51.658634 27068 raft_consensus.cc:2932] T f6920bfd97624ad381fd72b688ab73f1 P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0163beb78bfd4d399a70ebacb8f9aa9c (127.22.108.4) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "0163beb78bfd4d399a70ebacb8f9aa9c" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 45401 } } }
I20240503 02:11:51.660089 26315 raft_consensus.cc:2932] T f6920bfd97624ad381fd72b688ab73f1 P 0163beb78bfd4d399a70ebacb8f9aa9c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0163beb78bfd4d399a70ebacb8f9aa9c (127.22.108.4) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "0163beb78bfd4d399a70ebacb8f9aa9c" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 45401 } } }
I20240503 02:11:51.661805 27068 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:57988 (request call id 0) took 1063 ms. Trace:
I20240503 02:11:51.662012 27068 rpcz_store.cc:276] 0503 02:11:50.598611 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:11:50.598680 (+ 69us) service_pool.cc:225] Handling call
0503 02:11:50.599669 (+ 989us) log.cc:842] Serialized 257 byte log entry
0503 02:11:51.661753 (+1062084us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":41,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":231,"spinlock_wait_cycles":8960,"thread_start_us":186,"threads_started":1,"wal-append.queue_time_us":277,"wal-append.run_cpu_time_us":562,"wal-append.run_wall_time_us":1000586}
I20240503 02:11:51.664112 24846 catalog_manager.cc:5484] T f6920bfd97624ad381fd72b688ab73f1 P 0163beb78bfd4d399a70ebacb8f9aa9c reported cstate change: config changed from index -1 to 4, VOTER 0163beb78bfd4d399a70ebacb8f9aa9c (127.22.108.4) added. New cstate: current_term: 1 leader_uuid: "308436a6f7c7456b99971c666d3bd049" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "0163beb78bfd4d399a70ebacb8f9aa9c" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 45401 } } }
I20240503 02:11:51.694311 25121 consensus_queue.cc:237] T 4e5d560763774c0cba004753269a158c P 22b0b1ae6c504301ad5170c6bdcd179e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } }
W20240503 02:11:51.696264 25056 consensus_peers.cc:485] T 4e5d560763774c0cba004753269a158c P 22b0b1ae6c504301ad5170c6bdcd179e -> Peer 6c704dfc19814db996a1c0e363e760ef (127.22.108.6:43731): Couldn't send request to peer 6c704dfc19814db996a1c0e363e760ef. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e5d560763774c0cba004753269a158c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:52.142329 27093 ts_tablet_manager.cc:902] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef: Initiating tablet copy from peer 22b0b1ae6c504301ad5170c6bdcd179e (127.22.108.2:36213)
I20240503 02:11:52.143466 27093 tablet_copy_client.cc:308] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.2:36213
I20240503 02:11:52.144178 25141 tablet_copy_service.cc:140] P 22b0b1ae6c504301ad5170c6bdcd179e: Received BeginTabletCopySession request for tablet 4e5d560763774c0cba004753269a158c from peer 6c704dfc19814db996a1c0e363e760ef ({username='slave'} at 127.22.108.6:45763)
I20240503 02:11:52.144371 25141 tablet_copy_service.cc:161] P 22b0b1ae6c504301ad5170c6bdcd179e: Beginning new tablet copy session on tablet 4e5d560763774c0cba004753269a158c from peer 6c704dfc19814db996a1c0e363e760ef at {username='slave'} at 127.22.108.6:45763: session id = 6c704dfc19814db996a1c0e363e760ef-4e5d560763774c0cba004753269a158c
I20240503 02:11:52.146351 25141 tablet_copy_source_session.cc:205] T 4e5d560763774c0cba004753269a158c P 22b0b1ae6c504301ad5170c6bdcd179e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:11:52.147524 27093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e5d560763774c0cba004753269a158c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:52.151610 27093 tablet_copy_client.cc:783] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef: tablet copy: Starting download of 0 data blocks...
I20240503 02:11:52.151872 27093 tablet_copy_client.cc:647] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef: tablet copy: Starting download of 1 WAL segments...
I20240503 02:11:52.153507 27093 tablet_copy_client.cc:523] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:11:52.155745 27093 tablet_bootstrap.cc:492] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef: Bootstrap starting.
I20240503 02:11:52.186314 27093 tablet_bootstrap.cc:492] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 02:11:52.186702 27093 tablet_bootstrap.cc:492] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef: Bootstrap complete.
I20240503 02:11:52.186964 27093 ts_tablet_manager.cc:1367] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef: Time spent bootstrapping tablet: real 0.031s user 0.027s sys 0.004s
I20240503 02:11:52.187791 27093 raft_consensus.cc:348] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } }
I20240503 02:11:52.188228 27093 raft_consensus.cc:729] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c704dfc19814db996a1c0e363e760ef, State: Initialized, Role: FOLLOWER
I20240503 02:11:52.188411 27093 consensus_queue.cc:260] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } }
I20240503 02:11:52.189085 27093 ts_tablet_manager.cc:1398] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 02:11:52.190007 25141 tablet_copy_service.cc:342] P 22b0b1ae6c504301ad5170c6bdcd179e: Request end of tablet copy session 6c704dfc19814db996a1c0e363e760ef-4e5d560763774c0cba004753269a158c received from {username='slave'} at 127.22.108.6:45763
I20240503 02:11:52.190148 25141 tablet_copy_service.cc:433] P 22b0b1ae6c504301ad5170c6bdcd179e: ending tablet copy session 6c704dfc19814db996a1c0e363e760ef-4e5d560763774c0cba004753269a158c on tablet 4e5d560763774c0cba004753269a158c with peer 6c704dfc19814db996a1c0e363e760ef
I20240503 02:11:52.604143 26575 raft_consensus.cc:1203] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:11:52.605512 27100 raft_consensus.cc:2932] T 4e5d560763774c0cba004753269a158c P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6c704dfc19814db996a1c0e363e760ef (127.22.108.6) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } } }
I20240503 02:11:52.606076 26575 raft_consensus.cc:2932] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6c704dfc19814db996a1c0e363e760ef (127.22.108.6) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } } }
I20240503 02:11:52.610991 24846 catalog_manager.cc:5484] T 4e5d560763774c0cba004753269a158c P 22b0b1ae6c504301ad5170c6bdcd179e reported cstate change: config changed from index -1 to 4, VOTER 6c704dfc19814db996a1c0e363e760ef (127.22.108.6) added. New cstate: current_term: 1 leader_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } } }
I20240503 02:11:52.647914 25121 consensus_queue.cc:237] T 4ce4d66aff0e48deb5513001f1d81963 P 22b0b1ae6c504301ad5170c6bdcd179e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "1b2a71b48d4e40679504230af08279e9" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 40741 } }
W20240503 02:11:52.655604 25054 consensus_peers.cc:485] T 4ce4d66aff0e48deb5513001f1d81963 P 22b0b1ae6c504301ad5170c6bdcd179e -> Peer 1b2a71b48d4e40679504230af08279e9 (127.22.108.5:40741): Couldn't send request to peer 1b2a71b48d4e40679504230af08279e9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4ce4d66aff0e48deb5513001f1d81963. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:53.268777 27125 ts_tablet_manager.cc:902] T 4ce4d66aff0e48deb5513001f1d81963 P 1b2a71b48d4e40679504230af08279e9: Initiating tablet copy from peer 22b0b1ae6c504301ad5170c6bdcd179e (127.22.108.2:36213)
I20240503 02:11:53.270068 27125 tablet_copy_client.cc:308] T 4ce4d66aff0e48deb5513001f1d81963 P 1b2a71b48d4e40679504230af08279e9: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.2:36213
I20240503 02:11:53.276748 25141 tablet_copy_service.cc:140] P 22b0b1ae6c504301ad5170c6bdcd179e: Received BeginTabletCopySession request for tablet 4ce4d66aff0e48deb5513001f1d81963 from peer 1b2a71b48d4e40679504230af08279e9 ({username='slave'} at 127.22.108.5:35651)
I20240503 02:11:53.276917 25141 tablet_copy_service.cc:161] P 22b0b1ae6c504301ad5170c6bdcd179e: Beginning new tablet copy session on tablet 4ce4d66aff0e48deb5513001f1d81963 from peer 1b2a71b48d4e40679504230af08279e9 at {username='slave'} at 127.22.108.5:35651: session id = 1b2a71b48d4e40679504230af08279e9-4ce4d66aff0e48deb5513001f1d81963
I20240503 02:11:53.278954 25141 tablet_copy_source_session.cc:205] T 4ce4d66aff0e48deb5513001f1d81963 P 22b0b1ae6c504301ad5170c6bdcd179e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:11:53.280206 27125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ce4d66aff0e48deb5513001f1d81963. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:53.284557 27125 tablet_copy_client.cc:783] T 4ce4d66aff0e48deb5513001f1d81963 P 1b2a71b48d4e40679504230af08279e9: tablet copy: Starting download of 0 data blocks...
I20240503 02:11:53.284794 27125 tablet_copy_client.cc:647] T 4ce4d66aff0e48deb5513001f1d81963 P 1b2a71b48d4e40679504230af08279e9: tablet copy: Starting download of 1 WAL segments...
I20240503 02:11:53.286728 27125 tablet_copy_client.cc:523] T 4ce4d66aff0e48deb5513001f1d81963 P 1b2a71b48d4e40679504230af08279e9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:11:53.289113 27125 tablet_bootstrap.cc:492] T 4ce4d66aff0e48deb5513001f1d81963 P 1b2a71b48d4e40679504230af08279e9: Bootstrap starting.
I20240503 02:11:53.320101 27125 tablet_bootstrap.cc:492] T 4ce4d66aff0e48deb5513001f1d81963 P 1b2a71b48d4e40679504230af08279e9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 02:11:53.321014 27125 tablet_bootstrap.cc:492] T 4ce4d66aff0e48deb5513001f1d81963 P 1b2a71b48d4e40679504230af08279e9: Bootstrap complete.
I20240503 02:11:53.321333 27125 ts_tablet_manager.cc:1367] T 4ce4d66aff0e48deb5513001f1d81963 P 1b2a71b48d4e40679504230af08279e9: Time spent bootstrapping tablet: real 0.032s user 0.026s sys 0.007s
I20240503 02:11:53.322154 27125 raft_consensus.cc:348] T 4ce4d66aff0e48deb5513001f1d81963 P 1b2a71b48d4e40679504230af08279e9 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } }
I20240503 02:11:53.322517 27125 raft_consensus.cc:729] T 4ce4d66aff0e48deb5513001f1d81963 P 1b2a71b48d4e40679504230af08279e9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b2a71b48d4e40679504230af08279e9, State: Initialized, Role: FOLLOWER
I20240503 02:11:53.322703 27125 consensus_queue.cc:260] T 4ce4d66aff0e48deb5513001f1d81963 P 1b2a71b48d4e40679504230af08279e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "1b2a71b48d4e40679504230af08279e9" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 40741 } }
I20240503 02:11:53.323419 27125 ts_tablet_manager.cc:1398] T 4ce4d66aff0e48deb5513001f1d81963 P 1b2a71b48d4e40679504230af08279e9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:11:53.324276 25141 tablet_copy_service.cc:342] P 22b0b1ae6c504301ad5170c6bdcd179e: Request end of tablet copy session 1b2a71b48d4e40679504230af08279e9-4ce4d66aff0e48deb5513001f1d81963 received from {username='slave'} at 127.22.108.5:35651
I20240503 02:11:53.324435 25141 tablet_copy_service.cc:433] P 22b0b1ae6c504301ad5170c6bdcd179e: ending tablet copy session 1b2a71b48d4e40679504230af08279e9-4ce4d66aff0e48deb5513001f1d81963 on tablet 4ce4d66aff0e48deb5513001f1d81963 with peer 1b2a71b48d4e40679504230af08279e9
I20240503 02:11:53.648180 26445 raft_consensus.cc:1203] T 4ce4d66aff0e48deb5513001f1d81963 P 1b2a71b48d4e40679504230af08279e9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:11:53.649713 27134 raft_consensus.cc:2932] T 4ce4d66aff0e48deb5513001f1d81963 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1b2a71b48d4e40679504230af08279e9 (127.22.108.5) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "1b2a71b48d4e40679504230af08279e9" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 40741 } } }
I20240503 02:11:53.650324 26445 raft_consensus.cc:2932] T 4ce4d66aff0e48deb5513001f1d81963 P 1b2a71b48d4e40679504230af08279e9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1b2a71b48d4e40679504230af08279e9 (127.22.108.5) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "1b2a71b48d4e40679504230af08279e9" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 40741 } } }
I20240503 02:11:53.652832 27134 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:48742 (request call id 0) took 1005 ms. Trace:
I20240503 02:11:53.652978 27134 rpcz_store.cc:276] 0503 02:11:52.647339 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:11:52.647434 (+ 95us) service_pool.cc:225] Handling call
0503 02:11:52.648566 (+ 1132us) log.cc:842] Serialized 257 byte log entry
0503 02:11:53.652771 (+1004205us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":73,"raft.queue_time_us":47,"raft.run_cpu_time_us":262,"raft.run_wall_time_us":261,"spinlock_wait_cycles":38272,"thread_start_us":183,"threads_started":1,"wal-append.queue_time_us":292,"wal-append.run_cpu_time_us":555,"wal-append.run_wall_time_us":1000606}
I20240503 02:11:53.655025 24848 catalog_manager.cc:5484] T 4ce4d66aff0e48deb5513001f1d81963 P 22b0b1ae6c504301ad5170c6bdcd179e reported cstate change: config changed from index -1 to 4, VOTER 1b2a71b48d4e40679504230af08279e9 (127.22.108.5) added. New cstate: current_term: 1 leader_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "1b2a71b48d4e40679504230af08279e9" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 40741 } } }
I20240503 02:11:53.688539 24990 consensus_queue.cc:237] T e1093d22a1dc4e0f81fd3df39bf03265 P 308436a6f7c7456b99971c666d3bd049 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "b95ea99cb72f49c6b42c7e8e293cfc27" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 36823 } }
W20240503 02:11:53.696018 24923 consensus_peers.cc:485] T e1093d22a1dc4e0f81fd3df39bf03265 P 308436a6f7c7456b99971c666d3bd049 -> Peer b95ea99cb72f49c6b42c7e8e293cfc27 (127.22.108.3:36823): Couldn't send request to peer b95ea99cb72f49c6b42c7e8e293cfc27. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e1093d22a1dc4e0f81fd3df39bf03265. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:54.179478 27162 ts_tablet_manager.cc:902] T e1093d22a1dc4e0f81fd3df39bf03265 P b95ea99cb72f49c6b42c7e8e293cfc27: Initiating tablet copy from peer 308436a6f7c7456b99971c666d3bd049 (127.22.108.1:40941)
I20240503 02:11:54.180769 27162 tablet_copy_client.cc:308] T e1093d22a1dc4e0f81fd3df39bf03265 P b95ea99cb72f49c6b42c7e8e293cfc27: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.1:40941
I20240503 02:11:54.186980 25010 tablet_copy_service.cc:140] P 308436a6f7c7456b99971c666d3bd049: Received BeginTabletCopySession request for tablet e1093d22a1dc4e0f81fd3df39bf03265 from peer b95ea99cb72f49c6b42c7e8e293cfc27 ({username='slave'} at 127.22.108.3:58663)
I20240503 02:11:54.187136 25010 tablet_copy_service.cc:161] P 308436a6f7c7456b99971c666d3bd049: Beginning new tablet copy session on tablet e1093d22a1dc4e0f81fd3df39bf03265 from peer b95ea99cb72f49c6b42c7e8e293cfc27 at {username='slave'} at 127.22.108.3:58663: session id = b95ea99cb72f49c6b42c7e8e293cfc27-e1093d22a1dc4e0f81fd3df39bf03265
I20240503 02:11:54.189152 25010 tablet_copy_source_session.cc:205] T e1093d22a1dc4e0f81fd3df39bf03265 P 308436a6f7c7456b99971c666d3bd049: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:11:54.191008 27162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1093d22a1dc4e0f81fd3df39bf03265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:54.195636 27162 tablet_copy_client.cc:783] T e1093d22a1dc4e0f81fd3df39bf03265 P b95ea99cb72f49c6b42c7e8e293cfc27: tablet copy: Starting download of 0 data blocks...
I20240503 02:11:54.195928 27162 tablet_copy_client.cc:647] T e1093d22a1dc4e0f81fd3df39bf03265 P b95ea99cb72f49c6b42c7e8e293cfc27: tablet copy: Starting download of 1 WAL segments...
I20240503 02:11:54.197484 27162 tablet_copy_client.cc:523] T e1093d22a1dc4e0f81fd3df39bf03265 P b95ea99cb72f49c6b42c7e8e293cfc27: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:11:54.200218 27162 tablet_bootstrap.cc:492] T e1093d22a1dc4e0f81fd3df39bf03265 P b95ea99cb72f49c6b42c7e8e293cfc27: Bootstrap starting.
I20240503 02:11:54.233605 27162 tablet_bootstrap.cc:492] T e1093d22a1dc4e0f81fd3df39bf03265 P b95ea99cb72f49c6b42c7e8e293cfc27: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 02:11:54.234014 27162 tablet_bootstrap.cc:492] T e1093d22a1dc4e0f81fd3df39bf03265 P b95ea99cb72f49c6b42c7e8e293cfc27: Bootstrap complete.
I20240503 02:11:54.234267 27162 ts_tablet_manager.cc:1367] T e1093d22a1dc4e0f81fd3df39bf03265 P b95ea99cb72f49c6b42c7e8e293cfc27: Time spent bootstrapping tablet: real 0.034s user 0.030s sys 0.006s
I20240503 02:11:54.235093 27162 raft_consensus.cc:348] T e1093d22a1dc4e0f81fd3df39bf03265 P b95ea99cb72f49c6b42c7e8e293cfc27 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:11:54.235456 27162 raft_consensus.cc:729] T e1093d22a1dc4e0f81fd3df39bf03265 P b95ea99cb72f49c6b42c7e8e293cfc27 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95ea99cb72f49c6b42c7e8e293cfc27, State: Initialized, Role: FOLLOWER
I20240503 02:11:54.235623 27162 consensus_queue.cc:260] T e1093d22a1dc4e0f81fd3df39bf03265 P b95ea99cb72f49c6b42c7e8e293cfc27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "b95ea99cb72f49c6b42c7e8e293cfc27" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 36823 } }
I20240503 02:11:54.236397 27162 ts_tablet_manager.cc:1398] T e1093d22a1dc4e0f81fd3df39bf03265 P b95ea99cb72f49c6b42c7e8e293cfc27: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:11:54.237335 25010 tablet_copy_service.cc:342] P 308436a6f7c7456b99971c666d3bd049: Request end of tablet copy session b95ea99cb72f49c6b42c7e8e293cfc27-e1093d22a1dc4e0f81fd3df39bf03265 received from {username='slave'} at 127.22.108.3:58663
I20240503 02:11:54.237504 25010 tablet_copy_service.cc:433] P 308436a6f7c7456b99971c666d3bd049: ending tablet copy session b95ea99cb72f49c6b42c7e8e293cfc27-e1093d22a1dc4e0f81fd3df39bf03265 on tablet e1093d22a1dc4e0f81fd3df39bf03265 with peer b95ea99cb72f49c6b42c7e8e293cfc27
I20240503 02:11:54.667112 26184 raft_consensus.cc:1203] T e1093d22a1dc4e0f81fd3df39bf03265 P b95ea99cb72f49c6b42c7e8e293cfc27 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:11:54.668370 27170 raft_consensus.cc:2932] T e1093d22a1dc4e0f81fd3df39bf03265 P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b95ea99cb72f49c6b42c7e8e293cfc27 (127.22.108.3) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "b95ea99cb72f49c6b42c7e8e293cfc27" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 36823 } } }
I20240503 02:11:54.668962 26184 raft_consensus.cc:2932] T e1093d22a1dc4e0f81fd3df39bf03265 P b95ea99cb72f49c6b42c7e8e293cfc27 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b95ea99cb72f49c6b42c7e8e293cfc27 (127.22.108.3) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "b95ea99cb72f49c6b42c7e8e293cfc27" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 36823 } } }
I20240503 02:11:54.673159 24848 catalog_manager.cc:5484] T e1093d22a1dc4e0f81fd3df39bf03265 P 308436a6f7c7456b99971c666d3bd049 reported cstate change: config changed from index -1 to 4, VOTER b95ea99cb72f49c6b42c7e8e293cfc27 (127.22.108.3) added. New cstate: current_term: 1 leader_uuid: "308436a6f7c7456b99971c666d3bd049" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "b95ea99cb72f49c6b42c7e8e293cfc27" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 36823 } } }
I20240503 02:11:54.781095 26295 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:54.807741 26425 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:54.808512 26685 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:54.814751 24970 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:11:54.827664 25100 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:11:54.875216 26555 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:54.882423 26164 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:54.926074 24990 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e1093d22a1dc4e0f81fd3df39bf03265"
dest_uuid: "308436a6f7c7456b99971c666d3bd049"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38502
I20240503 02:11:54.926409 24990 raft_consensus.cc:595] T e1093d22a1dc4e0f81fd3df39bf03265 P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:11:55.038985 26295 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:55.053061 26555 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:55.053625 26685 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:55.066386 24970 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:11:55.100032 26425 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:55.102254 25100 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:11:55.121618 26164 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:55.169168 27170 raft_consensus.cc:979] T e1093d22a1dc4e0f81fd3df39bf03265 P 308436a6f7c7456b99971c666d3bd049: : Instructing follower b95ea99cb72f49c6b42c7e8e293cfc27 to start an election
I20240503 02:11:55.169407 27170 raft_consensus.cc:1067] T e1093d22a1dc4e0f81fd3df39bf03265 P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Signalling peer b95ea99cb72f49c6b42c7e8e293cfc27 to start an election
I20240503 02:11:55.170365 26184 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e1093d22a1dc4e0f81fd3df39bf03265"
dest_uuid: "b95ea99cb72f49c6b42c7e8e293cfc27"
from {username='slave'} at 127.22.108.1:43049
I20240503 02:11:55.170624 26184 raft_consensus.cc:482] T e1093d22a1dc4e0f81fd3df39bf03265 P b95ea99cb72f49c6b42c7e8e293cfc27 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:11:55.170835 26184 raft_consensus.cc:3037] T e1093d22a1dc4e0f81fd3df39bf03265 P b95ea99cb72f49c6b42c7e8e293cfc27 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:55.173589 26184 raft_consensus.cc:504] T e1093d22a1dc4e0f81fd3df39bf03265 P b95ea99cb72f49c6b42c7e8e293cfc27 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "b95ea99cb72f49c6b42c7e8e293cfc27" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 36823 } }
I20240503 02:11:55.175279 26184 leader_election.cc:288] T e1093d22a1dc4e0f81fd3df39bf03265 P b95ea99cb72f49c6b42c7e8e293cfc27 [CANDIDATE]: Term 2 election: Requested vote from peers 308436a6f7c7456b99971c666d3bd049 (127.22.108.1:40941)
I20240503 02:11:55.175961 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1093d22a1dc4e0f81fd3df39bf03265" candidate_uuid: "b95ea99cb72f49c6b42c7e8e293cfc27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "308436a6f7c7456b99971c666d3bd049"
I20240503 02:11:55.176255 24990 raft_consensus.cc:3032] T e1093d22a1dc4e0f81fd3df39bf03265 P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:11:55.176337 24990 raft_consensus.cc:729] T e1093d22a1dc4e0f81fd3df39bf03265 P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 308436a6f7c7456b99971c666d3bd049, State: Running, Role: LEADER
I20240503 02:11:55.176628 24990 consensus_queue.cc:260] T e1093d22a1dc4e0f81fd3df39bf03265 P 308436a6f7c7456b99971c666d3bd049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "b95ea99cb72f49c6b42c7e8e293cfc27" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 36823 } }
I20240503 02:11:55.176978 24990 raft_consensus.cc:3037] T e1093d22a1dc4e0f81fd3df39bf03265 P 308436a6f7c7456b99971c666d3bd049 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:55.179705 24990 raft_consensus.cc:2454] T e1093d22a1dc4e0f81fd3df39bf03265 P 308436a6f7c7456b99971c666d3bd049 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95ea99cb72f49c6b42c7e8e293cfc27 in term 2.
I20240503 02:11:55.180259 26119 leader_election.cc:302] T e1093d22a1dc4e0f81fd3df39bf03265 P b95ea99cb72f49c6b42c7e8e293cfc27 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 308436a6f7c7456b99971c666d3bd049, b95ea99cb72f49c6b42c7e8e293cfc27; no voters:
I20240503 02:11:55.180886 27267 raft_consensus.cc:2781] T e1093d22a1dc4e0f81fd3df39bf03265 P b95ea99cb72f49c6b42c7e8e293cfc27 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:11:55.182248 27267 raft_consensus.cc:686] T e1093d22a1dc4e0f81fd3df39bf03265 P b95ea99cb72f49c6b42c7e8e293cfc27 [term 2 LEADER]: Becoming Leader. State: Replica: b95ea99cb72f49c6b42c7e8e293cfc27, State: Running, Role: LEADER
I20240503 02:11:55.182713 27267 consensus_queue.cc:237] T e1093d22a1dc4e0f81fd3df39bf03265 P b95ea99cb72f49c6b42c7e8e293cfc27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "b95ea99cb72f49c6b42c7e8e293cfc27" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 36823 } }
I20240503 02:11:55.186429 24846 catalog_manager.cc:5484] T e1093d22a1dc4e0f81fd3df39bf03265 P b95ea99cb72f49c6b42c7e8e293cfc27 reported cstate change: term changed from 1 to 2, leader changed from 308436a6f7c7456b99971c666d3bd049 (127.22.108.1) to b95ea99cb72f49c6b42c7e8e293cfc27 (127.22.108.3). New cstate: current_term: 2 leader_uuid: "b95ea99cb72f49c6b42c7e8e293cfc27" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "b95ea99cb72f49c6b42c7e8e293cfc27" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 36823 } } }
I20240503 02:11:55.266446 26295 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:55.285722 26685 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:55.297420 26425 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:55.298624 25100 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:11:55.299660 26555 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:55.317633 24970 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:11:55.340651 26164 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:55.391194 25121 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4ce4d66aff0e48deb5513001f1d81963"
dest_uuid: "22b0b1ae6c504301ad5170c6bdcd179e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50890
I20240503 02:11:55.391479 25121 raft_consensus.cc:595] T 4ce4d66aff0e48deb5513001f1d81963 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 LEADER]: Received request to transfer leadership
I20240503 02:11:55.490396 26295 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:55.491417 26685 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:55.508318 24970 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:11:55.511840 26425 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:55.513147 25100 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:11:55.529181 26555 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:55.531451 26164 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:55.564268 24990 raft_consensus.cc:1261] T e1093d22a1dc4e0f81fd3df39bf03265 P 308436a6f7c7456b99971c666d3bd049 [term 2 FOLLOWER]: Refusing update from remote peer b95ea99cb72f49c6b42c7e8e293cfc27: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:11:55.565311 27267 consensus_queue.cc:1035] T e1093d22a1dc4e0f81fd3df39bf03265 P b95ea99cb72f49c6b42c7e8e293cfc27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:11:55.586302 27334 raft_consensus.cc:979] T 4ce4d66aff0e48deb5513001f1d81963 P 22b0b1ae6c504301ad5170c6bdcd179e: : Instructing follower 1b2a71b48d4e40679504230af08279e9 to start an election
I20240503 02:11:55.586573 27334 raft_consensus.cc:1067] T 4ce4d66aff0e48deb5513001f1d81963 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 LEADER]: Signalling peer 1b2a71b48d4e40679504230af08279e9 to start an election
I20240503 02:11:55.587747 26445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4ce4d66aff0e48deb5513001f1d81963"
dest_uuid: "1b2a71b48d4e40679504230af08279e9"
from {username='slave'} at 127.22.108.2:43469
I20240503 02:11:55.588063 26445 raft_consensus.cc:482] T 4ce4d66aff0e48deb5513001f1d81963 P 1b2a71b48d4e40679504230af08279e9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:11:55.588266 26445 raft_consensus.cc:3037] T 4ce4d66aff0e48deb5513001f1d81963 P 1b2a71b48d4e40679504230af08279e9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:55.591835 26445 raft_consensus.cc:504] T 4ce4d66aff0e48deb5513001f1d81963 P 1b2a71b48d4e40679504230af08279e9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "1b2a71b48d4e40679504230af08279e9" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 40741 } }
I20240503 02:11:55.595064 26445 leader_election.cc:288] T 4ce4d66aff0e48deb5513001f1d81963 P 1b2a71b48d4e40679504230af08279e9 [CANDIDATE]: Term 2 election: Requested vote from peers 22b0b1ae6c504301ad5170c6bdcd179e (127.22.108.2:36213)
I20240503 02:11:55.595779 25121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ce4d66aff0e48deb5513001f1d81963" candidate_uuid: "1b2a71b48d4e40679504230af08279e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "22b0b1ae6c504301ad5170c6bdcd179e"
I20240503 02:11:55.596117 25121 raft_consensus.cc:3032] T 4ce4d66aff0e48deb5513001f1d81963 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:11:55.596195 25121 raft_consensus.cc:729] T 4ce4d66aff0e48deb5513001f1d81963 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22b0b1ae6c504301ad5170c6bdcd179e, State: Running, Role: LEADER
I20240503 02:11:55.596542 25121 consensus_queue.cc:260] T 4ce4d66aff0e48deb5513001f1d81963 P 22b0b1ae6c504301ad5170c6bdcd179e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "1b2a71b48d4e40679504230af08279e9" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 40741 } }
I20240503 02:11:55.597002 25121 raft_consensus.cc:3037] T 4ce4d66aff0e48deb5513001f1d81963 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:55.600337 25121 raft_consensus.cc:2454] T 4ce4d66aff0e48deb5513001f1d81963 P 22b0b1ae6c504301ad5170c6bdcd179e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b2a71b48d4e40679504230af08279e9 in term 2.
I20240503 02:11:55.601076 26378 leader_election.cc:302] T 4ce4d66aff0e48deb5513001f1d81963 P 1b2a71b48d4e40679504230af08279e9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1b2a71b48d4e40679504230af08279e9, 22b0b1ae6c504301ad5170c6bdcd179e; no voters:
I20240503 02:11:55.603669 27337 raft_consensus.cc:2781] T 4ce4d66aff0e48deb5513001f1d81963 P 1b2a71b48d4e40679504230af08279e9 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:11:55.606662 27337 raft_consensus.cc:686] T 4ce4d66aff0e48deb5513001f1d81963 P 1b2a71b48d4e40679504230af08279e9 [term 2 LEADER]: Becoming Leader. State: Replica: 1b2a71b48d4e40679504230af08279e9, State: Running, Role: LEADER
I20240503 02:11:55.607298 27337 consensus_queue.cc:237] T 4ce4d66aff0e48deb5513001f1d81963 P 1b2a71b48d4e40679504230af08279e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "1b2a71b48d4e40679504230af08279e9" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 40741 } }
I20240503 02:11:55.612581 24840 catalog_manager.cc:5484] T 4ce4d66aff0e48deb5513001f1d81963 P 1b2a71b48d4e40679504230af08279e9 reported cstate change: term changed from 1 to 2, leader changed from 22b0b1ae6c504301ad5170c6bdcd179e (127.22.108.2) to 1b2a71b48d4e40679504230af08279e9 (127.22.108.5). New cstate: current_term: 2 leader_uuid: "1b2a71b48d4e40679504230af08279e9" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "1b2a71b48d4e40679504230af08279e9" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 40741 } } }
I20240503 02:11:55.722110 26295 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:55.723706 26425 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:55.727501 26685 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:55.733646 26555 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:55.747045 26164 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:55.752169 24970 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:11:55.764295 25100 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:55.836679 25121 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4e5d560763774c0cba004753269a158c"
dest_uuid: "22b0b1ae6c504301ad5170c6bdcd179e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50924
I20240503 02:11:55.836927 25121 raft_consensus.cc:595] T 4e5d560763774c0cba004753269a158c P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 LEADER]: Received request to transfer leadership
I20240503 02:11:55.942595 26295 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:55.946367 26685 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:55.952412 25100 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:55.952797 24970 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:11:55.954453 26425 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:55.962023 26555 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:55.990731 26164 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:56.094216 25121 raft_consensus.cc:1261] T 4ce4d66aff0e48deb5513001f1d81963 P 22b0b1ae6c504301ad5170c6bdcd179e [term 2 FOLLOWER]: Refusing update from remote peer 1b2a71b48d4e40679504230af08279e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:11:56.095194 27337 consensus_queue.cc:1035] T 4ce4d66aff0e48deb5513001f1d81963 P 1b2a71b48d4e40679504230af08279e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:11:56.165388 26295 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:56.168263 26685 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:56.186370 26425 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:56.199944 25100 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:56.206298 26555 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:56.213352 24970 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:11:56.225780 27454 raft_consensus.cc:979] T 4e5d560763774c0cba004753269a158c P 22b0b1ae6c504301ad5170c6bdcd179e: : Instructing follower 6c704dfc19814db996a1c0e363e760ef to start an election
I20240503 02:11:56.225972 27454 raft_consensus.cc:1067] T 4e5d560763774c0cba004753269a158c P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 LEADER]: Signalling peer 6c704dfc19814db996a1c0e363e760ef to start an election
I20240503 02:11:56.226224 26164 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:56.227093 26575 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e5d560763774c0cba004753269a158c"
dest_uuid: "6c704dfc19814db996a1c0e363e760ef"
from {username='slave'} at 127.22.108.2:51731
I20240503 02:11:56.227429 26575 raft_consensus.cc:482] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:11:56.227648 26575 raft_consensus.cc:3037] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:56.231132 26575 raft_consensus.cc:504] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } }
I20240503 02:11:56.233714 25121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5d560763774c0cba004753269a158c" candidate_uuid: "6c704dfc19814db996a1c0e363e760ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "22b0b1ae6c504301ad5170c6bdcd179e"
I20240503 02:11:56.233940 25121 raft_consensus.cc:3032] T 4e5d560763774c0cba004753269a158c P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:11:56.234031 25121 raft_consensus.cc:729] T 4e5d560763774c0cba004753269a158c P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22b0b1ae6c504301ad5170c6bdcd179e, State: Running, Role: LEADER
I20240503 02:11:56.234278 25121 consensus_queue.cc:260] T 4e5d560763774c0cba004753269a158c P 22b0b1ae6c504301ad5170c6bdcd179e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } }
I20240503 02:11:56.234508 25121 raft_consensus.cc:3037] T 4e5d560763774c0cba004753269a158c P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:56.238719 25121 raft_consensus.cc:2454] T 4e5d560763774c0cba004753269a158c P 22b0b1ae6c504301ad5170c6bdcd179e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c704dfc19814db996a1c0e363e760ef in term 2.
I20240503 02:11:56.239396 26508 leader_election.cc:302] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 22b0b1ae6c504301ad5170c6bdcd179e, 6c704dfc19814db996a1c0e363e760ef; no voters:
I20240503 02:11:56.232748 26575 leader_election.cc:288] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef [CANDIDATE]: Term 2 election: Requested vote from peers 22b0b1ae6c504301ad5170c6bdcd179e (127.22.108.2:36213)
I20240503 02:11:56.245849 27455 raft_consensus.cc:2781] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:11:56.271015 27455 raft_consensus.cc:686] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef [term 2 LEADER]: Becoming Leader. State: Replica: 6c704dfc19814db996a1c0e363e760ef, State: Running, Role: LEADER
I20240503 02:11:56.271632 27455 consensus_queue.cc:237] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } }
I20240503 02:11:56.287678 24845 catalog_manager.cc:5484] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef reported cstate change: term changed from 1 to 2, leader changed from 22b0b1ae6c504301ad5170c6bdcd179e (127.22.108.2) to 6c704dfc19814db996a1c0e363e760ef (127.22.108.6). New cstate: current_term: 2 leader_uuid: "6c704dfc19814db996a1c0e363e760ef" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } } }
I20240503 02:11:56.317361 24990 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f6920bfd97624ad381fd72b688ab73f1"
dest_uuid: "308436a6f7c7456b99971c666d3bd049"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38584
I20240503 02:11:56.317567 24990 raft_consensus.cc:595] T f6920bfd97624ad381fd72b688ab73f1 P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:11:56.432976 26295 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:56.434808 26425 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:56.443706 24970 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:11:56.447371 26685 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:56.461107 26555 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:56.461789 25100 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:56.479707 26164 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:56.638612 26295 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:56.647949 26425 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:56.656080 26685 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:56.689226 25100 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:56.689306 24970 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:11:56.692382 26555 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:56.714280 27535 raft_consensus.cc:979] T f6920bfd97624ad381fd72b688ab73f1 P 308436a6f7c7456b99971c666d3bd049: : Instructing follower 0163beb78bfd4d399a70ebacb8f9aa9c to start an election
I20240503 02:11:56.714704 27170 raft_consensus.cc:1067] T f6920bfd97624ad381fd72b688ab73f1 P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Signalling peer 0163beb78bfd4d399a70ebacb8f9aa9c to start an election
I20240503 02:11:56.715664 26314 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f6920bfd97624ad381fd72b688ab73f1"
dest_uuid: "0163beb78bfd4d399a70ebacb8f9aa9c"
from {username='slave'} at 127.22.108.1:56925
I20240503 02:11:56.715962 26314 raft_consensus.cc:482] T f6920bfd97624ad381fd72b688ab73f1 P 0163beb78bfd4d399a70ebacb8f9aa9c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:11:56.716732 26314 raft_consensus.cc:3037] T f6920bfd97624ad381fd72b688ab73f1 P 0163beb78bfd4d399a70ebacb8f9aa9c [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:56.719985 26314 raft_consensus.cc:504] T f6920bfd97624ad381fd72b688ab73f1 P 0163beb78bfd4d399a70ebacb8f9aa9c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "0163beb78bfd4d399a70ebacb8f9aa9c" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 45401 } }
I20240503 02:11:56.722728 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6920bfd97624ad381fd72b688ab73f1" candidate_uuid: "0163beb78bfd4d399a70ebacb8f9aa9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "308436a6f7c7456b99971c666d3bd049"
I20240503 02:11:56.722976 24990 raft_consensus.cc:3032] T f6920bfd97624ad381fd72b688ab73f1 P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:11:56.723086 24990 raft_consensus.cc:729] T f6920bfd97624ad381fd72b688ab73f1 P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 308436a6f7c7456b99971c666d3bd049, State: Running, Role: LEADER
I20240503 02:11:56.723384 24990 consensus_queue.cc:260] T f6920bfd97624ad381fd72b688ab73f1 P 308436a6f7c7456b99971c666d3bd049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "0163beb78bfd4d399a70ebacb8f9aa9c" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 45401 } }
I20240503 02:11:56.723915 26314 leader_election.cc:288] T f6920bfd97624ad381fd72b688ab73f1 P 0163beb78bfd4d399a70ebacb8f9aa9c [CANDIDATE]: Term 2 election: Requested vote from peers 308436a6f7c7456b99971c666d3bd049 (127.22.108.1:40941)
I20240503 02:11:56.724154 24990 raft_consensus.cc:3037] T f6920bfd97624ad381fd72b688ab73f1 P 308436a6f7c7456b99971c666d3bd049 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:56.727671 24990 raft_consensus.cc:2454] T f6920bfd97624ad381fd72b688ab73f1 P 308436a6f7c7456b99971c666d3bd049 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0163beb78bfd4d399a70ebacb8f9aa9c in term 2.
I20240503 02:11:56.728420 26250 leader_election.cc:302] T f6920bfd97624ad381fd72b688ab73f1 P 0163beb78bfd4d399a70ebacb8f9aa9c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0163beb78bfd4d399a70ebacb8f9aa9c, 308436a6f7c7456b99971c666d3bd049; no voters:
I20240503 02:11:56.739534 25121 raft_consensus.cc:1261] T 4e5d560763774c0cba004753269a158c P 22b0b1ae6c504301ad5170c6bdcd179e [term 2 FOLLOWER]: Refusing update from remote peer 6c704dfc19814db996a1c0e363e760ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:11:56.740749 27455 consensus_queue.cc:1035] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:11:56.744105 26164 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:56.749292 27536 raft_consensus.cc:2781] T f6920bfd97624ad381fd72b688ab73f1 P 0163beb78bfd4d399a70ebacb8f9aa9c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:11:56.759013 27536 raft_consensus.cc:686] T f6920bfd97624ad381fd72b688ab73f1 P 0163beb78bfd4d399a70ebacb8f9aa9c [term 2 LEADER]: Becoming Leader. State: Replica: 0163beb78bfd4d399a70ebacb8f9aa9c, State: Running, Role: LEADER
I20240503 02:11:56.759763 27536 consensus_queue.cc:237] T f6920bfd97624ad381fd72b688ab73f1 P 0163beb78bfd4d399a70ebacb8f9aa9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "0163beb78bfd4d399a70ebacb8f9aa9c" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 45401 } }
I20240503 02:11:56.766752 24845 catalog_manager.cc:5484] T f6920bfd97624ad381fd72b688ab73f1 P 0163beb78bfd4d399a70ebacb8f9aa9c reported cstate change: term changed from 1 to 2, leader changed from 308436a6f7c7456b99971c666d3bd049 (127.22.108.1) to 0163beb78bfd4d399a70ebacb8f9aa9c (127.22.108.4). New cstate: current_term: 2 leader_uuid: "0163beb78bfd4d399a70ebacb8f9aa9c" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "0163beb78bfd4d399a70ebacb8f9aa9c" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 45401 } } }
I20240503 02:11:56.796378 25121 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7b671abb263d4848b7b909927b7df380"
dest_uuid: "22b0b1ae6c504301ad5170c6bdcd179e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50992
I20240503 02:11:56.796617 25121 raft_consensus.cc:595] T 7b671abb263d4848b7b909927b7df380 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 LEADER]: Received request to transfer leadership
I20240503 02:11:56.902745 26295 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:56.905938 26685 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:56.911821 26425 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:56.940054 25100 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:56.956658 24970 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:11:56.967576 26555 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:57.001101 26164 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:57.152683 24990 raft_consensus.cc:1261] T f6920bfd97624ad381fd72b688ab73f1 P 308436a6f7c7456b99971c666d3bd049 [term 2 FOLLOWER]: Refusing update from remote peer 0163beb78bfd4d399a70ebacb8f9aa9c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:11:57.157508 27536 consensus_queue.cc:1035] T f6920bfd97624ad381fd72b688ab73f1 P 0163beb78bfd4d399a70ebacb8f9aa9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:11:57.186033 26555 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:57.190112 26425 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:57.191222 26295 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:57.195272 26685 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:57.195679 24970 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:11:57.195840 25100 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:57.198680 27613 raft_consensus.cc:979] T 7b671abb263d4848b7b909927b7df380 P 22b0b1ae6c504301ad5170c6bdcd179e: : Instructing follower 03afb03fe005462eb07ada37fe23c3cf to start an election
I20240503 02:11:57.198855 27613 raft_consensus.cc:1067] T 7b671abb263d4848b7b909927b7df380 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 LEADER]: Signalling peer 03afb03fe005462eb07ada37fe23c3cf to start an election
I20240503 02:11:57.199659 26705 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7b671abb263d4848b7b909927b7df380"
dest_uuid: "03afb03fe005462eb07ada37fe23c3cf"
from {username='slave'} at 127.22.108.2:41595
I20240503 02:11:57.199985 26705 raft_consensus.cc:482] T 7b671abb263d4848b7b909927b7df380 P 03afb03fe005462eb07ada37fe23c3cf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:11:57.200172 26705 raft_consensus.cc:3037] T 7b671abb263d4848b7b909927b7df380 P 03afb03fe005462eb07ada37fe23c3cf [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:57.203181 26705 raft_consensus.cc:504] T 7b671abb263d4848b7b909927b7df380 P 03afb03fe005462eb07ada37fe23c3cf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "03afb03fe005462eb07ada37fe23c3cf" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 34851 } }
I20240503 02:11:57.204998 26705 leader_election.cc:288] T 7b671abb263d4848b7b909927b7df380 P 03afb03fe005462eb07ada37fe23c3cf [CANDIDATE]: Term 2 election: Requested vote from peers 22b0b1ae6c504301ad5170c6bdcd179e (127.22.108.2:36213)
I20240503 02:11:57.206908 25121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b671abb263d4848b7b909927b7df380" candidate_uuid: "03afb03fe005462eb07ada37fe23c3cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "22b0b1ae6c504301ad5170c6bdcd179e"
I20240503 02:11:57.210352 25121 raft_consensus.cc:3032] T 7b671abb263d4848b7b909927b7df380 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:11:57.210469 25121 raft_consensus.cc:729] T 7b671abb263d4848b7b909927b7df380 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22b0b1ae6c504301ad5170c6bdcd179e, State: Running, Role: LEADER
I20240503 02:11:57.210759 25121 consensus_queue.cc:260] T 7b671abb263d4848b7b909927b7df380 P 22b0b1ae6c504301ad5170c6bdcd179e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "03afb03fe005462eb07ada37fe23c3cf" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 34851 } }
I20240503 02:11:57.211135 25121 raft_consensus.cc:3037] T 7b671abb263d4848b7b909927b7df380 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:57.213382 25121 raft_consensus.cc:2454] T 7b671abb263d4848b7b909927b7df380 P 22b0b1ae6c504301ad5170c6bdcd179e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03afb03fe005462eb07ada37fe23c3cf in term 2.
I20240503 02:11:57.214735 26638 leader_election.cc:302] T 7b671abb263d4848b7b909927b7df380 P 03afb03fe005462eb07ada37fe23c3cf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 03afb03fe005462eb07ada37fe23c3cf, 22b0b1ae6c504301ad5170c6bdcd179e; no voters:
I20240503 02:11:57.225862 27619 raft_consensus.cc:2781] T 7b671abb263d4848b7b909927b7df380 P 03afb03fe005462eb07ada37fe23c3cf [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:11:57.227449 26164 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:57.238471 27619 raft_consensus.cc:686] T 7b671abb263d4848b7b909927b7df380 P 03afb03fe005462eb07ada37fe23c3cf [term 2 LEADER]: Becoming Leader. State: Replica: 03afb03fe005462eb07ada37fe23c3cf, State: Running, Role: LEADER
I20240503 02:11:57.239148 27619 consensus_queue.cc:237] T 7b671abb263d4848b7b909927b7df380 P 03afb03fe005462eb07ada37fe23c3cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "03afb03fe005462eb07ada37fe23c3cf" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 34851 } }
I20240503 02:11:57.249680 24845 catalog_manager.cc:5484] T 7b671abb263d4848b7b909927b7df380 P 03afb03fe005462eb07ada37fe23c3cf reported cstate change: term changed from 1 to 2, leader changed from 22b0b1ae6c504301ad5170c6bdcd179e (127.22.108.2) to 03afb03fe005462eb07ada37fe23c3cf (127.22.108.7). New cstate: current_term: 2 leader_uuid: "03afb03fe005462eb07ada37fe23c3cf" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "03afb03fe005462eb07ada37fe23c3cf" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 34851 } } }
I20240503 02:11:57.299845 25121 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bb1a9efdfd96421ea17c41cfa8977f36"
dest_uuid: "22b0b1ae6c504301ad5170c6bdcd179e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51020
I20240503 02:11:57.300094 25121 raft_consensus.cc:595] T bb1a9efdfd96421ea17c41cfa8977f36 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 LEADER]: Received request to transfer leadership
I20240503 02:11:57.391953 26295 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:57.395118 26685 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:57.399488 26425 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:57.414767 24970 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:11:57.421736 26555 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:57.422430 25100 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:11:57.446710 26164 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:57.572678 27454 raft_consensus.cc:979] T bb1a9efdfd96421ea17c41cfa8977f36 P 22b0b1ae6c504301ad5170c6bdcd179e: : Instructing follower 6c704dfc19814db996a1c0e363e760ef to start an election
I20240503 02:11:57.572857 27454 raft_consensus.cc:1067] T bb1a9efdfd96421ea17c41cfa8977f36 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 LEADER]: Signalling peer 6c704dfc19814db996a1c0e363e760ef to start an election
I20240503 02:11:57.573525 26575 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bb1a9efdfd96421ea17c41cfa8977f36"
dest_uuid: "6c704dfc19814db996a1c0e363e760ef"
from {username='slave'} at 127.22.108.2:51731
I20240503 02:11:57.573740 26575 raft_consensus.cc:482] T bb1a9efdfd96421ea17c41cfa8977f36 P 6c704dfc19814db996a1c0e363e760ef [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:11:57.573881 26575 raft_consensus.cc:3037] T bb1a9efdfd96421ea17c41cfa8977f36 P 6c704dfc19814db996a1c0e363e760ef [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:57.577421 26575 raft_consensus.cc:504] T bb1a9efdfd96421ea17c41cfa8977f36 P 6c704dfc19814db996a1c0e363e760ef [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } }
I20240503 02:11:57.578037 26575 leader_election.cc:288] T bb1a9efdfd96421ea17c41cfa8977f36 P 6c704dfc19814db996a1c0e363e760ef [CANDIDATE]: Term 2 election: Requested vote from peers 22b0b1ae6c504301ad5170c6bdcd179e (127.22.108.2:36213)
I20240503 02:11:57.578507 25121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb1a9efdfd96421ea17c41cfa8977f36" candidate_uuid: "6c704dfc19814db996a1c0e363e760ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "22b0b1ae6c504301ad5170c6bdcd179e"
I20240503 02:11:57.578743 25121 raft_consensus.cc:3032] T bb1a9efdfd96421ea17c41cfa8977f36 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:11:57.578830 25121 raft_consensus.cc:729] T bb1a9efdfd96421ea17c41cfa8977f36 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22b0b1ae6c504301ad5170c6bdcd179e, State: Running, Role: LEADER
I20240503 02:11:57.579133 25121 consensus_queue.cc:260] T bb1a9efdfd96421ea17c41cfa8977f36 P 22b0b1ae6c504301ad5170c6bdcd179e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } }
I20240503 02:11:57.579298 25121 raft_consensus.cc:3037] T bb1a9efdfd96421ea17c41cfa8977f36 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:57.581821 25121 raft_consensus.cc:2454] T bb1a9efdfd96421ea17c41cfa8977f36 P 22b0b1ae6c504301ad5170c6bdcd179e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c704dfc19814db996a1c0e363e760ef in term 2.
I20240503 02:11:57.582226 26508 leader_election.cc:302] T bb1a9efdfd96421ea17c41cfa8977f36 P 6c704dfc19814db996a1c0e363e760ef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 22b0b1ae6c504301ad5170c6bdcd179e, 6c704dfc19814db996a1c0e363e760ef; no voters:
I20240503 02:11:57.582576 27455 raft_consensus.cc:2781] T bb1a9efdfd96421ea17c41cfa8977f36 P 6c704dfc19814db996a1c0e363e760ef [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:11:57.583628 27455 raft_consensus.cc:686] T bb1a9efdfd96421ea17c41cfa8977f36 P 6c704dfc19814db996a1c0e363e760ef [term 2 LEADER]: Becoming Leader. State: Replica: 6c704dfc19814db996a1c0e363e760ef, State: Running, Role: LEADER
I20240503 02:11:57.583951 27455 consensus_queue.cc:237] T bb1a9efdfd96421ea17c41cfa8977f36 P 6c704dfc19814db996a1c0e363e760ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } }
I20240503 02:11:57.586205 24845 catalog_manager.cc:5484] T bb1a9efdfd96421ea17c41cfa8977f36 P 6c704dfc19814db996a1c0e363e760ef reported cstate change: term changed from 1 to 2, leader changed from 22b0b1ae6c504301ad5170c6bdcd179e (127.22.108.2) to 6c704dfc19814db996a1c0e363e760ef (127.22.108.6). New cstate: current_term: 2 leader_uuid: "6c704dfc19814db996a1c0e363e760ef" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } } }
I20240503 02:11:57.611795 26295 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:57.639290 26685 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:57.640138 26425 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:57.658689 26164 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:57.667876 24970 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:11:57.683024 25100 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:57.685570 25120 raft_consensus.cc:1261] T 7b671abb263d4848b7b909927b7df380 P 22b0b1ae6c504301ad5170c6bdcd179e [term 2 FOLLOWER]: Refusing update from remote peer 03afb03fe005462eb07ada37fe23c3cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:11:57.687572 27619 consensus_queue.cc:1035] T 7b671abb263d4848b7b909927b7df380 P 03afb03fe005462eb07ada37fe23c3cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:11:57.703804 26555 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:57.768318 24990 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "923672905ffb452993c5dcf283f4a399"
dest_uuid: "308436a6f7c7456b99971c666d3bd049"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38670
I20240503 02:11:57.768560 24990 raft_consensus.cc:595] T 923672905ffb452993c5dcf283f4a399 P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:11:57.798342 27716 raft_consensus.cc:979] T 923672905ffb452993c5dcf283f4a399 P 308436a6f7c7456b99971c666d3bd049: : Instructing follower 1b2a71b48d4e40679504230af08279e9 to start an election
I20240503 02:11:57.798532 27716 raft_consensus.cc:1067] T 923672905ffb452993c5dcf283f4a399 P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Signalling peer 1b2a71b48d4e40679504230af08279e9 to start an election
I20240503 02:11:57.799252 26445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "923672905ffb452993c5dcf283f4a399"
dest_uuid: "1b2a71b48d4e40679504230af08279e9"
from {username='slave'} at 127.22.108.1:47109
I20240503 02:11:57.799470 26445 raft_consensus.cc:482] T 923672905ffb452993c5dcf283f4a399 P 1b2a71b48d4e40679504230af08279e9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:11:57.799572 26445 raft_consensus.cc:3037] T 923672905ffb452993c5dcf283f4a399 P 1b2a71b48d4e40679504230af08279e9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:57.801776 26445 raft_consensus.cc:504] T 923672905ffb452993c5dcf283f4a399 P 1b2a71b48d4e40679504230af08279e9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "1b2a71b48d4e40679504230af08279e9" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 40741 } }
I20240503 02:11:57.802297 26445 leader_election.cc:288] T 923672905ffb452993c5dcf283f4a399 P 1b2a71b48d4e40679504230af08279e9 [CANDIDATE]: Term 2 election: Requested vote from peers 308436a6f7c7456b99971c666d3bd049 (127.22.108.1:40941)
I20240503 02:11:57.802829 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "923672905ffb452993c5dcf283f4a399" candidate_uuid: "1b2a71b48d4e40679504230af08279e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "308436a6f7c7456b99971c666d3bd049"
I20240503 02:11:57.803036 24990 raft_consensus.cc:3032] T 923672905ffb452993c5dcf283f4a399 P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:11:57.803104 24990 raft_consensus.cc:729] T 923672905ffb452993c5dcf283f4a399 P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 308436a6f7c7456b99971c666d3bd049, State: Running, Role: LEADER
I20240503 02:11:57.803364 24990 consensus_queue.cc:260] T 923672905ffb452993c5dcf283f4a399 P 308436a6f7c7456b99971c666d3bd049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "1b2a71b48d4e40679504230af08279e9" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 40741 } }
I20240503 02:11:57.803637 24990 raft_consensus.cc:3037] T 923672905ffb452993c5dcf283f4a399 P 308436a6f7c7456b99971c666d3bd049 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:57.805626 24990 raft_consensus.cc:2454] T 923672905ffb452993c5dcf283f4a399 P 308436a6f7c7456b99971c666d3bd049 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b2a71b48d4e40679504230af08279e9 in term 2.
I20240503 02:11:57.806087 26380 leader_election.cc:302] T 923672905ffb452993c5dcf283f4a399 P 1b2a71b48d4e40679504230af08279e9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1b2a71b48d4e40679504230af08279e9, 308436a6f7c7456b99971c666d3bd049; no voters:
I20240503 02:11:57.806383 27691 raft_consensus.cc:2781] T 923672905ffb452993c5dcf283f4a399 P 1b2a71b48d4e40679504230af08279e9 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:11:57.806813 27691 raft_consensus.cc:686] T 923672905ffb452993c5dcf283f4a399 P 1b2a71b48d4e40679504230af08279e9 [term 2 LEADER]: Becoming Leader. State: Replica: 1b2a71b48d4e40679504230af08279e9, State: Running, Role: LEADER
I20240503 02:11:57.807158 27691 consensus_queue.cc:237] T 923672905ffb452993c5dcf283f4a399 P 1b2a71b48d4e40679504230af08279e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "1b2a71b48d4e40679504230af08279e9" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 40741 } }
I20240503 02:11:57.810206 24846 catalog_manager.cc:5484] T 923672905ffb452993c5dcf283f4a399 P 1b2a71b48d4e40679504230af08279e9 reported cstate change: term changed from 1 to 2, leader changed from 308436a6f7c7456b99971c666d3bd049 (127.22.108.1) to 1b2a71b48d4e40679504230af08279e9 (127.22.108.5). New cstate: current_term: 2 leader_uuid: "1b2a71b48d4e40679504230af08279e9" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "1b2a71b48d4e40679504230af08279e9" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 40741 } } }
I20240503 02:11:57.860019 26685 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:57.860209 26295 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:57.875106 25100 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:57.893252 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:57.896100 26555 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:57.911634 26164 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:57.913728 24970 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:57.976636 25120 raft_consensus.cc:1261] T bb1a9efdfd96421ea17c41cfa8977f36 P 22b0b1ae6c504301ad5170c6bdcd179e [term 2 FOLLOWER]: Refusing update from remote peer 6c704dfc19814db996a1c0e363e760ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:11:57.977316 27455 consensus_queue.cc:1035] T bb1a9efdfd96421ea17c41cfa8977f36 P 6c704dfc19814db996a1c0e363e760ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:11:58.088652 26295 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:58.102458 26685 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:58.120985 25100 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:58.123029 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:58.159798 24970 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:58.178544 26555 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:58.191537 26164 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:58.211184 24990 raft_consensus.cc:1261] T 923672905ffb452993c5dcf283f4a399 P 308436a6f7c7456b99971c666d3bd049 [term 2 FOLLOWER]: Refusing update from remote peer 1b2a71b48d4e40679504230af08279e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:11:58.211970 27691 consensus_queue.cc:1035] T 923672905ffb452993c5dcf283f4a399 P 1b2a71b48d4e40679504230af08279e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:11:58.242368 25120 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ee1a68079bfa435ab820967fb894d3f1"
dest_uuid: "22b0b1ae6c504301ad5170c6bdcd179e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51068
I20240503 02:11:58.242602 25120 raft_consensus.cc:595] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 LEADER]: Received request to transfer leadership
I20240503 02:11:58.285372 27799 raft_consensus.cc:979] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e: : Instructing follower b95ea99cb72f49c6b42c7e8e293cfc27 to start an election
I20240503 02:11:58.285573 27799 raft_consensus.cc:1067] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 LEADER]: Signalling peer b95ea99cb72f49c6b42c7e8e293cfc27 to start an election
I20240503 02:11:58.286382 26184 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ee1a68079bfa435ab820967fb894d3f1"
dest_uuid: "b95ea99cb72f49c6b42c7e8e293cfc27"
from {username='slave'} at 127.22.108.2:44049
I20240503 02:11:58.286655 26184 raft_consensus.cc:482] T ee1a68079bfa435ab820967fb894d3f1 P b95ea99cb72f49c6b42c7e8e293cfc27 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:11:58.286783 26184 raft_consensus.cc:3037] T ee1a68079bfa435ab820967fb894d3f1 P b95ea99cb72f49c6b42c7e8e293cfc27 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:58.289225 26184 raft_consensus.cc:504] T ee1a68079bfa435ab820967fb894d3f1 P b95ea99cb72f49c6b42c7e8e293cfc27 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "b95ea99cb72f49c6b42c7e8e293cfc27" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 36823 } }
I20240503 02:11:58.289891 26184 leader_election.cc:288] T ee1a68079bfa435ab820967fb894d3f1 P b95ea99cb72f49c6b42c7e8e293cfc27 [CANDIDATE]: Term 2 election: Requested vote from peers 22b0b1ae6c504301ad5170c6bdcd179e (127.22.108.2:36213)
I20240503 02:11:58.290477 25120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee1a68079bfa435ab820967fb894d3f1" candidate_uuid: "b95ea99cb72f49c6b42c7e8e293cfc27" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "22b0b1ae6c504301ad5170c6bdcd179e"
I20240503 02:11:58.290736 25120 raft_consensus.cc:3032] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:11:58.290848 25120 raft_consensus.cc:729] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 22b0b1ae6c504301ad5170c6bdcd179e, State: Running, Role: LEADER
I20240503 02:11:58.291133 25120 consensus_queue.cc:260] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "b95ea99cb72f49c6b42c7e8e293cfc27" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 36823 } }
I20240503 02:11:58.291486 25120 raft_consensus.cc:3037] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:58.293741 25120 raft_consensus.cc:2454] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95ea99cb72f49c6b42c7e8e293cfc27 in term 2.
I20240503 02:11:58.294234 26117 leader_election.cc:302] T ee1a68079bfa435ab820967fb894d3f1 P b95ea99cb72f49c6b42c7e8e293cfc27 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 22b0b1ae6c504301ad5170c6bdcd179e, b95ea99cb72f49c6b42c7e8e293cfc27; no voters:
I20240503 02:11:58.294705 27688 raft_consensus.cc:2781] T ee1a68079bfa435ab820967fb894d3f1 P b95ea99cb72f49c6b42c7e8e293cfc27 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:11:58.295150 27688 raft_consensus.cc:686] T ee1a68079bfa435ab820967fb894d3f1 P b95ea99cb72f49c6b42c7e8e293cfc27 [term 2 LEADER]: Becoming Leader. State: Replica: b95ea99cb72f49c6b42c7e8e293cfc27, State: Running, Role: LEADER
I20240503 02:11:58.295466 27688 consensus_queue.cc:237] T ee1a68079bfa435ab820967fb894d3f1 P b95ea99cb72f49c6b42c7e8e293cfc27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "b95ea99cb72f49c6b42c7e8e293cfc27" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 36823 } }
I20240503 02:11:58.298890 24846 catalog_manager.cc:5484] T ee1a68079bfa435ab820967fb894d3f1 P b95ea99cb72f49c6b42c7e8e293cfc27 reported cstate change: term changed from 1 to 2, leader changed from 22b0b1ae6c504301ad5170c6bdcd179e (127.22.108.2) to b95ea99cb72f49c6b42c7e8e293cfc27 (127.22.108.3). New cstate: current_term: 2 leader_uuid: "b95ea99cb72f49c6b42c7e8e293cfc27" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 } } peers { permanent_uuid: "b95ea99cb72f49c6b42c7e8e293cfc27" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 36823 } } }
I20240503 02:11:58.347211 26295 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:58.348389 26685 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:58.375435 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:58.376174 24970 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:58.386147 25100 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:58.414117 26164 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:58.416860 26555 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:58.593173 25100 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:58.594324 24970 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:58.595191 26685 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:58.597333 26295 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:58.607911 26555 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:58.608192 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:58.611073 26164 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:58.700809 24990 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7c0b16db9b77442b922e3b7bf1247878"
dest_uuid: "308436a6f7c7456b99971c666d3bd049"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38726
I20240503 02:11:58.701069 24990 raft_consensus.cc:595] T 7c0b16db9b77442b922e3b7bf1247878 P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:11:58.795850 25120 raft_consensus.cc:1261] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e [term 2 FOLLOWER]: Refusing update from remote peer b95ea99cb72f49c6b42c7e8e293cfc27: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:11:58.796509 26295 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:58.797000 27688 consensus_queue.cc:1035] T ee1a68079bfa435ab820967fb894d3f1 P b95ea99cb72f49c6b42c7e8e293cfc27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b0b1ae6c504301ad5170c6bdcd179e" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 36213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:11:58.798777 26685 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:58.852583 24970 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:11:58.890219 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:58.893996 26555 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:58.910234 25100 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:58.925596 26164 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:58.977734 27906 raft_consensus.cc:979] T 7c0b16db9b77442b922e3b7bf1247878 P 308436a6f7c7456b99971c666d3bd049: : Instructing follower 0163beb78bfd4d399a70ebacb8f9aa9c to start an election
I20240503 02:11:58.977948 27906 raft_consensus.cc:1067] T 7c0b16db9b77442b922e3b7bf1247878 P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Signalling peer 0163beb78bfd4d399a70ebacb8f9aa9c to start an election
I20240503 02:11:58.978683 26314 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7c0b16db9b77442b922e3b7bf1247878"
dest_uuid: "0163beb78bfd4d399a70ebacb8f9aa9c"
from {username='slave'} at 127.22.108.1:56925
I20240503 02:11:58.978901 26314 raft_consensus.cc:482] T 7c0b16db9b77442b922e3b7bf1247878 P 0163beb78bfd4d399a70ebacb8f9aa9c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:11:58.979000 26314 raft_consensus.cc:3037] T 7c0b16db9b77442b922e3b7bf1247878 P 0163beb78bfd4d399a70ebacb8f9aa9c [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:58.980948 26314 raft_consensus.cc:504] T 7c0b16db9b77442b922e3b7bf1247878 P 0163beb78bfd4d399a70ebacb8f9aa9c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "0163beb78bfd4d399a70ebacb8f9aa9c" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 45401 } }
I20240503 02:11:58.981492 26314 leader_election.cc:288] T 7c0b16db9b77442b922e3b7bf1247878 P 0163beb78bfd4d399a70ebacb8f9aa9c [CANDIDATE]: Term 2 election: Requested vote from peers 308436a6f7c7456b99971c666d3bd049 (127.22.108.1:40941)
I20240503 02:11:58.981986 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c0b16db9b77442b922e3b7bf1247878" candidate_uuid: "0163beb78bfd4d399a70ebacb8f9aa9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "308436a6f7c7456b99971c666d3bd049"
I20240503 02:11:58.982203 24990 raft_consensus.cc:3032] T 7c0b16db9b77442b922e3b7bf1247878 P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:11:58.982306 24990 raft_consensus.cc:729] T 7c0b16db9b77442b922e3b7bf1247878 P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 308436a6f7c7456b99971c666d3bd049, State: Running, Role: LEADER
I20240503 02:11:58.982563 24990 consensus_queue.cc:260] T 7c0b16db9b77442b922e3b7bf1247878 P 308436a6f7c7456b99971c666d3bd049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "0163beb78bfd4d399a70ebacb8f9aa9c" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 45401 } }
I20240503 02:11:58.982839 24990 raft_consensus.cc:3037] T 7c0b16db9b77442b922e3b7bf1247878 P 308436a6f7c7456b99971c666d3bd049 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:58.985569 24990 raft_consensus.cc:2454] T 7c0b16db9b77442b922e3b7bf1247878 P 308436a6f7c7456b99971c666d3bd049 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0163beb78bfd4d399a70ebacb8f9aa9c in term 2.
I20240503 02:11:58.986011 26250 leader_election.cc:302] T 7c0b16db9b77442b922e3b7bf1247878 P 0163beb78bfd4d399a70ebacb8f9aa9c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0163beb78bfd4d399a70ebacb8f9aa9c, 308436a6f7c7456b99971c666d3bd049; no voters:
I20240503 02:11:58.986418 27898 raft_consensus.cc:2781] T 7c0b16db9b77442b922e3b7bf1247878 P 0163beb78bfd4d399a70ebacb8f9aa9c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:11:58.986575 27898 raft_consensus.cc:686] T 7c0b16db9b77442b922e3b7bf1247878 P 0163beb78bfd4d399a70ebacb8f9aa9c [term 2 LEADER]: Becoming Leader. State: Replica: 0163beb78bfd4d399a70ebacb8f9aa9c, State: Running, Role: LEADER
I20240503 02:11:58.986912 27898 consensus_queue.cc:237] T 7c0b16db9b77442b922e3b7bf1247878 P 0163beb78bfd4d399a70ebacb8f9aa9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "0163beb78bfd4d399a70ebacb8f9aa9c" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 45401 } }
I20240503 02:11:58.997525 24840 catalog_manager.cc:5484] T 7c0b16db9b77442b922e3b7bf1247878 P 0163beb78bfd4d399a70ebacb8f9aa9c reported cstate change: term changed from 1 to 2, leader changed from 308436a6f7c7456b99971c666d3bd049 (127.22.108.1) to 0163beb78bfd4d399a70ebacb8f9aa9c (127.22.108.4). New cstate: current_term: 2 leader_uuid: "0163beb78bfd4d399a70ebacb8f9aa9c" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "0163beb78bfd4d399a70ebacb8f9aa9c" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 45401 } } }
I20240503 02:11:59.102802 26685 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:59.104393 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:59.117940 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:59.126813 26555 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:59.131561 24970 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:59.154716 25100 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:59.177810 26164 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:59.235656 24990 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ed43f867d21d4bee9efc0b9bf4ee90c1"
dest_uuid: "308436a6f7c7456b99971c666d3bd049"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38768
I20240503 02:11:59.235906 24990 raft_consensus.cc:595] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:11:59.253718 27716 raft_consensus.cc:979] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 308436a6f7c7456b99971c666d3bd049: : Instructing follower 03afb03fe005462eb07ada37fe23c3cf to start an election
I20240503 02:11:59.253918 27716 raft_consensus.cc:1067] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Signalling peer 03afb03fe005462eb07ada37fe23c3cf to start an election
I20240503 02:11:59.254546 26705 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ed43f867d21d4bee9efc0b9bf4ee90c1"
dest_uuid: "03afb03fe005462eb07ada37fe23c3cf"
from {username='slave'} at 127.22.108.1:34577
I20240503 02:11:59.254776 26705 raft_consensus.cc:482] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 03afb03fe005462eb07ada37fe23c3cf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:11:59.254882 26705 raft_consensus.cc:3037] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 03afb03fe005462eb07ada37fe23c3cf [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:59.257284 26705 raft_consensus.cc:504] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 03afb03fe005462eb07ada37fe23c3cf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "03afb03fe005462eb07ada37fe23c3cf" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 34851 } }
I20240503 02:11:59.257826 26705 leader_election.cc:288] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 03afb03fe005462eb07ada37fe23c3cf [CANDIDATE]: Term 2 election: Requested vote from peers 308436a6f7c7456b99971c666d3bd049 (127.22.108.1:40941)
I20240503 02:11:59.258423 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed43f867d21d4bee9efc0b9bf4ee90c1" candidate_uuid: "03afb03fe005462eb07ada37fe23c3cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "308436a6f7c7456b99971c666d3bd049"
I20240503 02:11:59.258611 24990 raft_consensus.cc:3032] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:11:59.258718 24990 raft_consensus.cc:729] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 308436a6f7c7456b99971c666d3bd049, State: Running, Role: LEADER
I20240503 02:11:59.258977 24990 consensus_queue.cc:260] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 308436a6f7c7456b99971c666d3bd049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "03afb03fe005462eb07ada37fe23c3cf" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 34851 } }
I20240503 02:11:59.259245 24990 raft_consensus.cc:3037] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 308436a6f7c7456b99971c666d3bd049 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:59.261397 24990 raft_consensus.cc:2454] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 308436a6f7c7456b99971c666d3bd049 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03afb03fe005462eb07ada37fe23c3cf in term 2.
I20240503 02:11:59.261785 26640 leader_election.cc:302] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 03afb03fe005462eb07ada37fe23c3cf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 03afb03fe005462eb07ada37fe23c3cf, 308436a6f7c7456b99971c666d3bd049; no voters:
I20240503 02:11:59.262076 27900 raft_consensus.cc:2781] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 03afb03fe005462eb07ada37fe23c3cf [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:11:59.262234 27900 raft_consensus.cc:686] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 03afb03fe005462eb07ada37fe23c3cf [term 2 LEADER]: Becoming Leader. State: Replica: 03afb03fe005462eb07ada37fe23c3cf, State: Running, Role: LEADER
I20240503 02:11:59.262531 27900 consensus_queue.cc:237] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 03afb03fe005462eb07ada37fe23c3cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "03afb03fe005462eb07ada37fe23c3cf" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 34851 } }
I20240503 02:11:59.265129 24848 catalog_manager.cc:5484] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 03afb03fe005462eb07ada37fe23c3cf reported cstate change: term changed from 1 to 2, leader changed from 308436a6f7c7456b99971c666d3bd049 (127.22.108.1) to 03afb03fe005462eb07ada37fe23c3cf (127.22.108.7). New cstate: current_term: 2 leader_uuid: "03afb03fe005462eb07ada37fe23c3cf" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "03afb03fe005462eb07ada37fe23c3cf" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 34851 } } }
I20240503 02:11:59.340008 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:59.352573 26685 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:59.371532 24970 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:11:59.375474 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:59.387161 26555 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:59.392683 25100 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:59.403815 26164 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:59.598568 24990 raft_consensus.cc:1261] T 7c0b16db9b77442b922e3b7bf1247878 P 308436a6f7c7456b99971c666d3bd049 [term 2 FOLLOWER]: Refusing update from remote peer 0163beb78bfd4d399a70ebacb8f9aa9c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:11:59.599228 27898 consensus_queue.cc:1035] T 7c0b16db9b77442b922e3b7bf1247878 P 0163beb78bfd4d399a70ebacb8f9aa9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:11:59.751677 24990 raft_consensus.cc:1261] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 308436a6f7c7456b99971c666d3bd049 [term 2 FOLLOWER]: Refusing update from remote peer 03afb03fe005462eb07ada37fe23c3cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:11:59.752559 27900 consensus_queue.cc:1035] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 03afb03fe005462eb07ada37fe23c3cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:11:59.778061 26685 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:59.782948 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:59.801766 24970 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:11:59.822527 26555 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:59.825829 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:59.826851 25100 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:59.858222 26164 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:59.946367 26184 consensus_queue.cc:237] T e1093d22a1dc4e0f81fd3df39bf03265 P b95ea99cb72f49c6b42c7e8e293cfc27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b95ea99cb72f49c6b42c7e8e293cfc27" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 36823 } }
I20240503 02:11:59.947670 27901 raft_consensus.cc:2932] T e1093d22a1dc4e0f81fd3df39bf03265 P b95ea99cb72f49c6b42c7e8e293cfc27 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 308436a6f7c7456b99971c666d3bd049 (127.22.108.1) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b95ea99cb72f49c6b42c7e8e293cfc27" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 36823 } } }
I20240503 02:11:59.953085 24840 catalog_manager.cc:5484] T e1093d22a1dc4e0f81fd3df39bf03265 P b95ea99cb72f49c6b42c7e8e293cfc27 reported cstate change: config changed from index 4 to 6, VOTER 308436a6f7c7456b99971c666d3bd049 (127.22.108.1) evicted. New cstate: current_term: 2 leader_uuid: "b95ea99cb72f49c6b42c7e8e293cfc27" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b95ea99cb72f49c6b42c7e8e293cfc27" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 36823 } } }
I20240503 02:11:59.958422 24970 tablet_service.cc:1514] Processing DeleteTablet for tablet e1093d22a1dc4e0f81fd3df39bf03265 with delete_type TABLET_DATA_TOMBSTONED (TS 308436a6f7c7456b99971c666d3bd049 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47868
I20240503 02:11:59.962782 28052 tablet_replica.cc:330] T e1093d22a1dc4e0f81fd3df39bf03265 P 308436a6f7c7456b99971c666d3bd049: stopping tablet replica
I20240503 02:11:59.963304 28052 raft_consensus.cc:2229] T e1093d22a1dc4e0f81fd3df39bf03265 P 308436a6f7c7456b99971c666d3bd049 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:11:59.963550 28052 raft_consensus.cc:2258] T e1093d22a1dc4e0f81fd3df39bf03265 P 308436a6f7c7456b99971c666d3bd049 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:11:59.964965 28052 ts_tablet_manager.cc:1875] T e1093d22a1dc4e0f81fd3df39bf03265 P 308436a6f7c7456b99971c666d3bd049: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:11:59.971462 28052 ts_tablet_manager.cc:1888] T e1093d22a1dc4e0f81fd3df39bf03265 P 308436a6f7c7456b99971c666d3bd049: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 02:11:59.971668 28052 log.cc:1196] T e1093d22a1dc4e0f81fd3df39bf03265 P 308436a6f7c7456b99971c666d3bd049: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/e1093d22a1dc4e0f81fd3df39bf03265
I20240503 02:11:59.972582 24835 catalog_manager.cc:4830] TS 308436a6f7c7456b99971c666d3bd049 (127.22.108.1:40941): tablet e1093d22a1dc4e0f81fd3df39bf03265 (table rebalance_test_table_0 [id=adca86e19a68413aa3c2e0c525d683a5]) successfully deleted
I20240503 02:12:00.062093 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:00.069310 26685 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:00.072194 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:00.080351 24970 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:00.080526 25100 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:00.105149 26555 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:00.131417 26164 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:00.207969 26445 consensus_queue.cc:237] T 4ce4d66aff0e48deb5513001f1d81963 P 1b2a71b48d4e40679504230af08279e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1b2a71b48d4e40679504230af08279e9" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 40741 } }
I20240503 02:12:00.209252 27691 raft_consensus.cc:2932] T 4ce4d66aff0e48deb5513001f1d81963 P 1b2a71b48d4e40679504230af08279e9 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 22b0b1ae6c504301ad5170c6bdcd179e (127.22.108.2) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1b2a71b48d4e40679504230af08279e9" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 40741 } } }
I20240503 02:12:00.214051 24844 catalog_manager.cc:5484] T 4ce4d66aff0e48deb5513001f1d81963 P 1b2a71b48d4e40679504230af08279e9 reported cstate change: config changed from index 4 to 6, VOTER 22b0b1ae6c504301ad5170c6bdcd179e (127.22.108.2) evicted. New cstate: current_term: 2 leader_uuid: "1b2a71b48d4e40679504230af08279e9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1b2a71b48d4e40679504230af08279e9" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 40741 } } }
I20240503 02:12:00.218001 25100 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ce4d66aff0e48deb5513001f1d81963 with delete_type TABLET_DATA_TOMBSTONED (TS 22b0b1ae6c504301ad5170c6bdcd179e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56630
I20240503 02:12:00.218884 28107 tablet_replica.cc:330] T 4ce4d66aff0e48deb5513001f1d81963 P 22b0b1ae6c504301ad5170c6bdcd179e: stopping tablet replica
I20240503 02:12:00.219300 28107 raft_consensus.cc:2229] T 4ce4d66aff0e48deb5513001f1d81963 P 22b0b1ae6c504301ad5170c6bdcd179e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:00.219569 28107 raft_consensus.cc:2258] T 4ce4d66aff0e48deb5513001f1d81963 P 22b0b1ae6c504301ad5170c6bdcd179e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:00.220748 28107 ts_tablet_manager.cc:1875] T 4ce4d66aff0e48deb5513001f1d81963 P 22b0b1ae6c504301ad5170c6bdcd179e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:00.226135 28107 ts_tablet_manager.cc:1888] T 4ce4d66aff0e48deb5513001f1d81963 P 22b0b1ae6c504301ad5170c6bdcd179e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 02:12:00.226300 28107 log.cc:1196] T 4ce4d66aff0e48deb5513001f1d81963 P 22b0b1ae6c504301ad5170c6bdcd179e: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/4ce4d66aff0e48deb5513001f1d81963
I20240503 02:12:00.227667 24833 catalog_manager.cc:4830] TS 22b0b1ae6c504301ad5170c6bdcd179e (127.22.108.2:36213): tablet 4ce4d66aff0e48deb5513001f1d81963 (table rebalance_test_table_1 [id=dcb3b303097f4c479078780f22a99899]) successfully deleted
I20240503 02:12:00.325073 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:00.325613 26685 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:00.359123 24970 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:00.360440 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:00.361207 25100 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:00.384864 26555 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:00.387534 26164 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:00.485173 26575 consensus_queue.cc:237] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } }
I20240503 02:12:00.486469 27686 raft_consensus.cc:2932] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 22b0b1ae6c504301ad5170c6bdcd179e (127.22.108.2) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } } }
I20240503 02:12:00.491681 24844 catalog_manager.cc:5484] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef reported cstate change: config changed from index 4 to 6, VOTER 22b0b1ae6c504301ad5170c6bdcd179e (127.22.108.2) evicted. New cstate: current_term: 2 leader_uuid: "6c704dfc19814db996a1c0e363e760ef" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } } }
I20240503 02:12:00.495541 25100 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e5d560763774c0cba004753269a158c with delete_type TABLET_DATA_TOMBSTONED (TS 22b0b1ae6c504301ad5170c6bdcd179e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56630
I20240503 02:12:00.495930 28107 tablet_replica.cc:330] T 4e5d560763774c0cba004753269a158c P 22b0b1ae6c504301ad5170c6bdcd179e: stopping tablet replica
I20240503 02:12:00.496184 28107 raft_consensus.cc:2229] T 4e5d560763774c0cba004753269a158c P 22b0b1ae6c504301ad5170c6bdcd179e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:00.496382 28107 raft_consensus.cc:2258] T 4e5d560763774c0cba004753269a158c P 22b0b1ae6c504301ad5170c6bdcd179e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:00.497176 28107 ts_tablet_manager.cc:1875] T 4e5d560763774c0cba004753269a158c P 22b0b1ae6c504301ad5170c6bdcd179e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:00.502604 28107 ts_tablet_manager.cc:1888] T 4e5d560763774c0cba004753269a158c P 22b0b1ae6c504301ad5170c6bdcd179e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 02:12:00.502743 28107 log.cc:1196] T 4e5d560763774c0cba004753269a158c P 22b0b1ae6c504301ad5170c6bdcd179e: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/4e5d560763774c0cba004753269a158c
I20240503 02:12:00.503528 24833 catalog_manager.cc:4830] TS 22b0b1ae6c504301ad5170c6bdcd179e (127.22.108.2:36213): tablet 4e5d560763774c0cba004753269a158c (table rebalance_test_table_3 [id=ac074aff80ae4832bb677cde4982aed7]) successfully deleted
I20240503 02:12:00.588662 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:00.603263 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:00.603864 26685 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:00.609865 26555 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:00.614965 24970 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:00.640323 26164 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:00.650096 25100 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:00.751663 26314 consensus_queue.cc:237] T f6920bfd97624ad381fd72b688ab73f1 P 0163beb78bfd4d399a70ebacb8f9aa9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0163beb78bfd4d399a70ebacb8f9aa9c" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 45401 } }
I20240503 02:12:00.752923 27985 raft_consensus.cc:2932] T f6920bfd97624ad381fd72b688ab73f1 P 0163beb78bfd4d399a70ebacb8f9aa9c [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 308436a6f7c7456b99971c666d3bd049 (127.22.108.1) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0163beb78bfd4d399a70ebacb8f9aa9c" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 45401 } } }
I20240503 02:12:00.757380 24848 catalog_manager.cc:5484] T f6920bfd97624ad381fd72b688ab73f1 P 0163beb78bfd4d399a70ebacb8f9aa9c reported cstate change: config changed from index 4 to 6, VOTER 308436a6f7c7456b99971c666d3bd049 (127.22.108.1) evicted. New cstate: current_term: 2 leader_uuid: "0163beb78bfd4d399a70ebacb8f9aa9c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0163beb78bfd4d399a70ebacb8f9aa9c" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 45401 } } }
I20240503 02:12:00.761377 24970 tablet_service.cc:1514] Processing DeleteTablet for tablet f6920bfd97624ad381fd72b688ab73f1 with delete_type TABLET_DATA_TOMBSTONED (TS 308436a6f7c7456b99971c666d3bd049 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47868
I20240503 02:12:00.762118 28215 tablet_replica.cc:330] T f6920bfd97624ad381fd72b688ab73f1 P 308436a6f7c7456b99971c666d3bd049: stopping tablet replica
I20240503 02:12:00.762382 28215 raft_consensus.cc:2229] T f6920bfd97624ad381fd72b688ab73f1 P 308436a6f7c7456b99971c666d3bd049 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:00.762671 28215 raft_consensus.cc:2258] T f6920bfd97624ad381fd72b688ab73f1 P 308436a6f7c7456b99971c666d3bd049 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:00.763386 28215 ts_tablet_manager.cc:1875] T f6920bfd97624ad381fd72b688ab73f1 P 308436a6f7c7456b99971c666d3bd049: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:00.769161 28215 ts_tablet_manager.cc:1888] T f6920bfd97624ad381fd72b688ab73f1 P 308436a6f7c7456b99971c666d3bd049: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 02:12:00.769297 28215 log.cc:1196] T f6920bfd97624ad381fd72b688ab73f1 P 308436a6f7c7456b99971c666d3bd049: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/f6920bfd97624ad381fd72b688ab73f1
I20240503 02:12:00.770157 24835 catalog_manager.cc:4830] TS 308436a6f7c7456b99971c666d3bd049 (127.22.108.1:40941): tablet f6920bfd97624ad381fd72b688ab73f1 (table rebalance_test_table_4 [id=83cdd3376f31411b849eb85a2660ae9f]) successfully deleted
I20240503 02:12:00.865639 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:00.873276 26685 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:00.874549 24970 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:00.876046 26555 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:00.884140 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:00.898237 26164 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:00.916096 25100 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:01.005357 26705 consensus_queue.cc:237] T 7b671abb263d4848b7b909927b7df380 P 03afb03fe005462eb07ada37fe23c3cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "03afb03fe005462eb07ada37fe23c3cf" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 34851 } }
I20240503 02:12:01.006629 27905 raft_consensus.cc:2932] T 7b671abb263d4848b7b909927b7df380 P 03afb03fe005462eb07ada37fe23c3cf [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 22b0b1ae6c504301ad5170c6bdcd179e (127.22.108.2) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "03afb03fe005462eb07ada37fe23c3cf" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 34851 } } }
I20240503 02:12:01.011993 24848 catalog_manager.cc:5484] T 7b671abb263d4848b7b909927b7df380 P 03afb03fe005462eb07ada37fe23c3cf reported cstate change: config changed from index 4 to 6, VOTER 22b0b1ae6c504301ad5170c6bdcd179e (127.22.108.2) evicted. New cstate: current_term: 2 leader_uuid: "03afb03fe005462eb07ada37fe23c3cf" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "03afb03fe005462eb07ada37fe23c3cf" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 34851 } } }
I20240503 02:12:01.016203 25100 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b671abb263d4848b7b909927b7df380 with delete_type TABLET_DATA_TOMBSTONED (TS 22b0b1ae6c504301ad5170c6bdcd179e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56630
I20240503 02:12:01.016914 28272 tablet_replica.cc:330] T 7b671abb263d4848b7b909927b7df380 P 22b0b1ae6c504301ad5170c6bdcd179e: stopping tablet replica
I20240503 02:12:01.017123 28272 raft_consensus.cc:2229] T 7b671abb263d4848b7b909927b7df380 P 22b0b1ae6c504301ad5170c6bdcd179e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:01.017318 28272 raft_consensus.cc:2258] T 7b671abb263d4848b7b909927b7df380 P 22b0b1ae6c504301ad5170c6bdcd179e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:01.017953 28272 ts_tablet_manager.cc:1875] T 7b671abb263d4848b7b909927b7df380 P 22b0b1ae6c504301ad5170c6bdcd179e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:01.022475 28272 ts_tablet_manager.cc:1888] T 7b671abb263d4848b7b909927b7df380 P 22b0b1ae6c504301ad5170c6bdcd179e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 02:12:01.022591 28272 log.cc:1196] T 7b671abb263d4848b7b909927b7df380 P 22b0b1ae6c504301ad5170c6bdcd179e: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/7b671abb263d4848b7b909927b7df380
I20240503 02:12:01.023268 24833 catalog_manager.cc:4830] TS 22b0b1ae6c504301ad5170c6bdcd179e (127.22.108.2:36213): tablet 7b671abb263d4848b7b909927b7df380 (table rebalance_test_table_3 [id=ac074aff80ae4832bb677cde4982aed7]) successfully deleted
I20240503 02:12:01.120487 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:01.135681 26685 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:01.143085 24970 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:01.166806 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:01.167117 26555 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:01.183140 25100 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:01.191653 26164 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:01.267896 26575 consensus_queue.cc:237] T bb1a9efdfd96421ea17c41cfa8977f36 P 6c704dfc19814db996a1c0e363e760ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } }
I20240503 02:12:01.269280 28279 raft_consensus.cc:2932] T bb1a9efdfd96421ea17c41cfa8977f36 P 6c704dfc19814db996a1c0e363e760ef [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 22b0b1ae6c504301ad5170c6bdcd179e (127.22.108.2) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } } }
I20240503 02:12:01.273799 24848 catalog_manager.cc:5484] T bb1a9efdfd96421ea17c41cfa8977f36 P 6c704dfc19814db996a1c0e363e760ef reported cstate change: config changed from index 4 to 6, VOTER 22b0b1ae6c504301ad5170c6bdcd179e (127.22.108.2) evicted. New cstate: current_term: 2 leader_uuid: "6c704dfc19814db996a1c0e363e760ef" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } } }
I20240503 02:12:01.277664 25100 tablet_service.cc:1514] Processing DeleteTablet for tablet bb1a9efdfd96421ea17c41cfa8977f36 with delete_type TABLET_DATA_TOMBSTONED (TS 22b0b1ae6c504301ad5170c6bdcd179e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56630
I20240503 02:12:01.278043 28272 tablet_replica.cc:330] T bb1a9efdfd96421ea17c41cfa8977f36 P 22b0b1ae6c504301ad5170c6bdcd179e: stopping tablet replica
I20240503 02:12:01.278255 28272 raft_consensus.cc:2229] T bb1a9efdfd96421ea17c41cfa8977f36 P 22b0b1ae6c504301ad5170c6bdcd179e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:01.278467 28272 raft_consensus.cc:2258] T bb1a9efdfd96421ea17c41cfa8977f36 P 22b0b1ae6c504301ad5170c6bdcd179e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:01.279099 28272 ts_tablet_manager.cc:1875] T bb1a9efdfd96421ea17c41cfa8977f36 P 22b0b1ae6c504301ad5170c6bdcd179e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:01.285980 28272 ts_tablet_manager.cc:1888] T bb1a9efdfd96421ea17c41cfa8977f36 P 22b0b1ae6c504301ad5170c6bdcd179e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 02:12:01.286111 28272 log.cc:1196] T bb1a9efdfd96421ea17c41cfa8977f36 P 22b0b1ae6c504301ad5170c6bdcd179e: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/bb1a9efdfd96421ea17c41cfa8977f36
I20240503 02:12:01.286991 24833 catalog_manager.cc:4830] TS 22b0b1ae6c504301ad5170c6bdcd179e (127.22.108.2:36213): tablet bb1a9efdfd96421ea17c41cfa8977f36 (table rebalance_test_table_1 [id=dcb3b303097f4c479078780f22a99899]) successfully deleted
I20240503 02:12:01.384071 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:01.386126 26685 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:01.391041 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:01.399070 26164 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:01.410790 24970 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:01.417198 25100 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:01.420936 26555 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:01.524225 26445 consensus_queue.cc:237] T 923672905ffb452993c5dcf283f4a399 P 1b2a71b48d4e40679504230af08279e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1b2a71b48d4e40679504230af08279e9" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 40741 } }
I20240503 02:12:01.526108 28360 raft_consensus.cc:2932] T 923672905ffb452993c5dcf283f4a399 P 1b2a71b48d4e40679504230af08279e9 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 308436a6f7c7456b99971c666d3bd049 (127.22.108.1) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1b2a71b48d4e40679504230af08279e9" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 40741 } } }
I20240503 02:12:01.530721 24846 catalog_manager.cc:5484] T 923672905ffb452993c5dcf283f4a399 P 1b2a71b48d4e40679504230af08279e9 reported cstate change: config changed from index 4 to 6, VOTER 308436a6f7c7456b99971c666d3bd049 (127.22.108.1) evicted. New cstate: current_term: 2 leader_uuid: "1b2a71b48d4e40679504230af08279e9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1b2a71b48d4e40679504230af08279e9" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 40741 } } }
I20240503 02:12:01.534392 24970 tablet_service.cc:1514] Processing DeleteTablet for tablet 923672905ffb452993c5dcf283f4a399 with delete_type TABLET_DATA_TOMBSTONED (TS 308436a6f7c7456b99971c666d3bd049 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47868
I20240503 02:12:01.535625 28383 tablet_replica.cc:330] T 923672905ffb452993c5dcf283f4a399 P 308436a6f7c7456b99971c666d3bd049: stopping tablet replica
I20240503 02:12:01.535827 28383 raft_consensus.cc:2229] T 923672905ffb452993c5dcf283f4a399 P 308436a6f7c7456b99971c666d3bd049 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:01.536065 28383 raft_consensus.cc:2258] T 923672905ffb452993c5dcf283f4a399 P 308436a6f7c7456b99971c666d3bd049 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:01.536713 28383 ts_tablet_manager.cc:1875] T 923672905ffb452993c5dcf283f4a399 P 308436a6f7c7456b99971c666d3bd049: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:01.542533 28383 ts_tablet_manager.cc:1888] T 923672905ffb452993c5dcf283f4a399 P 308436a6f7c7456b99971c666d3bd049: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 02:12:01.542703 28383 log.cc:1196] T 923672905ffb452993c5dcf283f4a399 P 308436a6f7c7456b99971c666d3bd049: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/923672905ffb452993c5dcf283f4a399
I20240503 02:12:01.543502 24835 catalog_manager.cc:4830] TS 308436a6f7c7456b99971c666d3bd049 (127.22.108.1:40941): tablet 923672905ffb452993c5dcf283f4a399 (table rebalance_test_table_4 [id=83cdd3376f31411b849eb85a2660ae9f]) successfully deleted
I20240503 02:12:01.631925 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:01.632771 26685 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:01.633525 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:01.658545 24970 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:01.658831 26164 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:01.659657 26555 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:01.685528 25100 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:01.771098 26184 consensus_queue.cc:237] T ee1a68079bfa435ab820967fb894d3f1 P b95ea99cb72f49c6b42c7e8e293cfc27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b95ea99cb72f49c6b42c7e8e293cfc27" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 36823 } }
I20240503 02:12:01.772473 27901 raft_consensus.cc:2932] T ee1a68079bfa435ab820967fb894d3f1 P b95ea99cb72f49c6b42c7e8e293cfc27 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 22b0b1ae6c504301ad5170c6bdcd179e (127.22.108.2) evicted. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b95ea99cb72f49c6b42c7e8e293cfc27" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 36823 } } }
I20240503 02:12:01.776849 24845 catalog_manager.cc:5484] T ee1a68079bfa435ab820967fb894d3f1 P b95ea99cb72f49c6b42c7e8e293cfc27 reported cstate change: config changed from index 2 to 4, VOTER 22b0b1ae6c504301ad5170c6bdcd179e (127.22.108.2) evicted. New cstate: current_term: 2 leader_uuid: "b95ea99cb72f49c6b42c7e8e293cfc27" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b95ea99cb72f49c6b42c7e8e293cfc27" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 36823 } } }
I20240503 02:12:01.781738 25100 tablet_service.cc:1514] Processing DeleteTablet for tablet ee1a68079bfa435ab820967fb894d3f1 with delete_type TABLET_DATA_TOMBSTONED (TS 22b0b1ae6c504301ad5170c6bdcd179e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56630
I20240503 02:12:01.782081 28272 tablet_replica.cc:330] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e: stopping tablet replica
I20240503 02:12:01.782266 28272 raft_consensus.cc:2229] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:01.782474 28272 raft_consensus.cc:2258] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:01.783103 28272 ts_tablet_manager.cc:1875] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:01.789611 28272 ts_tablet_manager.cc:1888] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240503 02:12:01.789827 28272 log.cc:1196] T ee1a68079bfa435ab820967fb894d3f1 P 22b0b1ae6c504301ad5170c6bdcd179e: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/ee1a68079bfa435ab820967fb894d3f1
I20240503 02:12:01.790648 24833 catalog_manager.cc:4830] TS 22b0b1ae6c504301ad5170c6bdcd179e (127.22.108.2:36213): tablet ee1a68079bfa435ab820967fb894d3f1 (table TestTable [id=152bc29db2bb47d98fa21091fc2aaf3d]) successfully deleted
I20240503 02:12:01.891155 26685 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:01.906000 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:01.908198 25100 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:01.918152 24970 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:01.930013 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:01.970714 26555 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:01.974654 26164 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:02.058012 26314 consensus_queue.cc:237] T 7c0b16db9b77442b922e3b7bf1247878 P 0163beb78bfd4d399a70ebacb8f9aa9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0163beb78bfd4d399a70ebacb8f9aa9c" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 45401 } }
I20240503 02:12:02.059341 28454 raft_consensus.cc:2932] T 7c0b16db9b77442b922e3b7bf1247878 P 0163beb78bfd4d399a70ebacb8f9aa9c [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 308436a6f7c7456b99971c666d3bd049 (127.22.108.1) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0163beb78bfd4d399a70ebacb8f9aa9c" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 45401 } } }
I20240503 02:12:02.063864 24844 catalog_manager.cc:5484] T 7c0b16db9b77442b922e3b7bf1247878 P 0163beb78bfd4d399a70ebacb8f9aa9c reported cstate change: config changed from index 4 to 6, VOTER 308436a6f7c7456b99971c666d3bd049 (127.22.108.1) evicted. New cstate: current_term: 2 leader_uuid: "0163beb78bfd4d399a70ebacb8f9aa9c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0163beb78bfd4d399a70ebacb8f9aa9c" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 45401 } } }
I20240503 02:12:02.067601 24970 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c0b16db9b77442b922e3b7bf1247878 with delete_type TABLET_DATA_TOMBSTONED (TS 308436a6f7c7456b99971c666d3bd049 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47868
I20240503 02:12:02.068310 28487 tablet_replica.cc:330] T 7c0b16db9b77442b922e3b7bf1247878 P 308436a6f7c7456b99971c666d3bd049: stopping tablet replica
I20240503 02:12:02.068550 28487 raft_consensus.cc:2229] T 7c0b16db9b77442b922e3b7bf1247878 P 308436a6f7c7456b99971c666d3bd049 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:02.068746 28487 raft_consensus.cc:2258] T 7c0b16db9b77442b922e3b7bf1247878 P 308436a6f7c7456b99971c666d3bd049 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:02.069427 28487 ts_tablet_manager.cc:1875] T 7c0b16db9b77442b922e3b7bf1247878 P 308436a6f7c7456b99971c666d3bd049: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:02.074704 28487 ts_tablet_manager.cc:1888] T 7c0b16db9b77442b922e3b7bf1247878 P 308436a6f7c7456b99971c666d3bd049: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 02:12:02.074855 28487 log.cc:1196] T 7c0b16db9b77442b922e3b7bf1247878 P 308436a6f7c7456b99971c666d3bd049: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/7c0b16db9b77442b922e3b7bf1247878
I20240503 02:12:02.075748 24835 catalog_manager.cc:4830] TS 308436a6f7c7456b99971c666d3bd049 (127.22.108.1:40941): tablet 7c0b16db9b77442b922e3b7bf1247878 (table rebalance_test_table_2 [id=7aa88098eef344a7aebdefe66f69aa8b]) successfully deleted
I20240503 02:12:02.169570 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:02.184650 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:02.186478 26685 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:02.186839 25100 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:02.187141 24970 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:02.193434 26555 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:02.203526 26164 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:02.303357 26705 consensus_queue.cc:237] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 03afb03fe005462eb07ada37fe23c3cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "03afb03fe005462eb07ada37fe23c3cf" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 34851 } }
I20240503 02:12:02.304642 27905 raft_consensus.cc:2932] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 03afb03fe005462eb07ada37fe23c3cf [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 308436a6f7c7456b99971c666d3bd049 (127.22.108.1) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "03afb03fe005462eb07ada37fe23c3cf" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 34851 } } }
I20240503 02:12:02.309356 24845 catalog_manager.cc:5484] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 03afb03fe005462eb07ada37fe23c3cf reported cstate change: config changed from index 4 to 6, VOTER 308436a6f7c7456b99971c666d3bd049 (127.22.108.1) evicted. New cstate: current_term: 2 leader_uuid: "03afb03fe005462eb07ada37fe23c3cf" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "03afb03fe005462eb07ada37fe23c3cf" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 34851 } } }
I20240503 02:12:02.312819 24970 tablet_service.cc:1514] Processing DeleteTablet for tablet ed43f867d21d4bee9efc0b9bf4ee90c1 with delete_type TABLET_DATA_TOMBSTONED (TS 308436a6f7c7456b99971c666d3bd049 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47868
I20240503 02:12:02.313180 28487 tablet_replica.cc:330] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 308436a6f7c7456b99971c666d3bd049: stopping tablet replica
I20240503 02:12:02.313335 28487 raft_consensus.cc:2229] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 308436a6f7c7456b99971c666d3bd049 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:02.313503 28487 raft_consensus.cc:2258] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 308436a6f7c7456b99971c666d3bd049 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:02.314064 28487 ts_tablet_manager.cc:1875] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 308436a6f7c7456b99971c666d3bd049: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:02.318295 28487 ts_tablet_manager.cc:1888] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 308436a6f7c7456b99971c666d3bd049: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 02:12:02.318393 28487 log.cc:1196] T ed43f867d21d4bee9efc0b9bf4ee90c1 P 308436a6f7c7456b99971c666d3bd049: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/ed43f867d21d4bee9efc0b9bf4ee90c1
I20240503 02:12:02.319000 24835 catalog_manager.cc:4830] TS 308436a6f7c7456b99971c666d3bd049 (127.22.108.1:40941): tablet ed43f867d21d4bee9efc0b9bf4ee90c1 (table rebalance_test_table_2 [id=7aa88098eef344a7aebdefe66f69aa8b]) successfully deleted
I20240503 02:12:02.628931 24990 consensus_queue.cc:237] T a141985c93da484494d5f82c478d0bc3 P 308436a6f7c7456b99971c666d3bd049 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } }
W20240503 02:12:02.638808 24925 consensus_peers.cc:485] T a141985c93da484494d5f82c478d0bc3 P 308436a6f7c7456b99971c666d3bd049 -> Peer 6c704dfc19814db996a1c0e363e760ef (127.22.108.6:43731): Couldn't send request to peer 6c704dfc19814db996a1c0e363e760ef. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a141985c93da484494d5f82c478d0bc3. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:03.209059 28610 ts_tablet_manager.cc:902] T a141985c93da484494d5f82c478d0bc3 P 6c704dfc19814db996a1c0e363e760ef: Initiating tablet copy from peer 308436a6f7c7456b99971c666d3bd049 (127.22.108.1:40941)
I20240503 02:12:03.210389 28610 tablet_copy_client.cc:308] T a141985c93da484494d5f82c478d0bc3 P 6c704dfc19814db996a1c0e363e760ef: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.1:40941
I20240503 02:12:03.217082 25010 tablet_copy_service.cc:140] P 308436a6f7c7456b99971c666d3bd049: Received BeginTabletCopySession request for tablet a141985c93da484494d5f82c478d0bc3 from peer 6c704dfc19814db996a1c0e363e760ef ({username='slave'} at 127.22.108.6:56075)
I20240503 02:12:03.217273 25010 tablet_copy_service.cc:161] P 308436a6f7c7456b99971c666d3bd049: Beginning new tablet copy session on tablet a141985c93da484494d5f82c478d0bc3 from peer 6c704dfc19814db996a1c0e363e760ef at {username='slave'} at 127.22.108.6:56075: session id = 6c704dfc19814db996a1c0e363e760ef-a141985c93da484494d5f82c478d0bc3
I20240503 02:12:03.219163 25010 tablet_copy_source_session.cc:205] T a141985c93da484494d5f82c478d0bc3 P 308436a6f7c7456b99971c666d3bd049: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:03.220389 28610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a141985c93da484494d5f82c478d0bc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:03.224970 28610 tablet_copy_client.cc:783] T a141985c93da484494d5f82c478d0bc3 P 6c704dfc19814db996a1c0e363e760ef: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:03.225255 28610 tablet_copy_client.cc:647] T a141985c93da484494d5f82c478d0bc3 P 6c704dfc19814db996a1c0e363e760ef: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:03.227028 28610 tablet_copy_client.cc:523] T a141985c93da484494d5f82c478d0bc3 P 6c704dfc19814db996a1c0e363e760ef: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:03.229503 28610 tablet_bootstrap.cc:492] T a141985c93da484494d5f82c478d0bc3 P 6c704dfc19814db996a1c0e363e760ef: Bootstrap starting.
I20240503 02:12:03.259570 28610 tablet_bootstrap.cc:492] T a141985c93da484494d5f82c478d0bc3 P 6c704dfc19814db996a1c0e363e760ef: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 02:12:03.259989 28610 tablet_bootstrap.cc:492] T a141985c93da484494d5f82c478d0bc3 P 6c704dfc19814db996a1c0e363e760ef: Bootstrap complete.
I20240503 02:12:03.260265 28610 ts_tablet_manager.cc:1367] T a141985c93da484494d5f82c478d0bc3 P 6c704dfc19814db996a1c0e363e760ef: Time spent bootstrapping tablet: real 0.031s user 0.023s sys 0.005s
I20240503 02:12:03.260960 28610 raft_consensus.cc:348] T a141985c93da484494d5f82c478d0bc3 P 6c704dfc19814db996a1c0e363e760ef [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:12:03.261392 28610 raft_consensus.cc:729] T a141985c93da484494d5f82c478d0bc3 P 6c704dfc19814db996a1c0e363e760ef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c704dfc19814db996a1c0e363e760ef, State: Initialized, Role: FOLLOWER
I20240503 02:12:03.261616 28610 consensus_queue.cc:260] T a141985c93da484494d5f82c478d0bc3 P 6c704dfc19814db996a1c0e363e760ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } }
I20240503 02:12:03.262270 28610 ts_tablet_manager.cc:1398] T a141985c93da484494d5f82c478d0bc3 P 6c704dfc19814db996a1c0e363e760ef: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240503 02:12:03.263149 25010 tablet_copy_service.cc:342] P 308436a6f7c7456b99971c666d3bd049: Request end of tablet copy session 6c704dfc19814db996a1c0e363e760ef-a141985c93da484494d5f82c478d0bc3 received from {username='slave'} at 127.22.108.6:56075
I20240503 02:12:03.263296 25010 tablet_copy_service.cc:433] P 308436a6f7c7456b99971c666d3bd049: ending tablet copy session 6c704dfc19814db996a1c0e363e760ef-a141985c93da484494d5f82c478d0bc3 on tablet a141985c93da484494d5f82c478d0bc3 with peer 6c704dfc19814db996a1c0e363e760ef
I20240503 02:12:03.813596 26575 raft_consensus.cc:1203] T a141985c93da484494d5f82c478d0bc3 P 6c704dfc19814db996a1c0e363e760ef [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:12:03.815268 28621 raft_consensus.cc:2932] T a141985c93da484494d5f82c478d0bc3 P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6c704dfc19814db996a1c0e363e760ef (127.22.108.6) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } } }
I20240503 02:12:03.815881 26575 raft_consensus.cc:2932] T a141985c93da484494d5f82c478d0bc3 P 6c704dfc19814db996a1c0e363e760ef [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6c704dfc19814db996a1c0e363e760ef (127.22.108.6) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } } }
I20240503 02:12:03.818609 28621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:38894 (request call id 0) took 1190 ms. Trace:
I20240503 02:12:03.818825 28621 rpcz_store.cc:276] 0503 02:12:02.628277 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:02.628348 (+ 71us) service_pool.cc:225] Handling call
0503 02:12:02.629661 (+ 1313us) log.cc:842] Serialized 257 byte log entry
0503 02:12:03.818557 (+1188896us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":61,"raft.queue_time_us":379,"raft.run_cpu_time_us":288,"raft.run_wall_time_us":288,"thread_start_us":1273,"threads_started":2,"wal-append.queue_time_us":1125,"wal-append.run_cpu_time_us":662,"wal-append.run_wall_time_us":1000709}
I20240503 02:12:03.820670 24843 catalog_manager.cc:5484] T a141985c93da484494d5f82c478d0bc3 P 308436a6f7c7456b99971c666d3bd049 reported cstate change: config changed from index -1 to 4, VOTER 6c704dfc19814db996a1c0e363e760ef (127.22.108.6) added. New cstate: current_term: 1 leader_uuid: "308436a6f7c7456b99971c666d3bd049" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } } }
I20240503 02:12:03.926352 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:03.936055 26685 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:03.939139 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:03.965735 24970 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:03.981252 26555 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:04.014818 25100 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:04.016844 26164 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:04.072723 24990 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a141985c93da484494d5f82c478d0bc3"
dest_uuid: "308436a6f7c7456b99971c666d3bd049"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39922
I20240503 02:12:04.072975 24990 raft_consensus.cc:595] T a141985c93da484494d5f82c478d0bc3 P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:12:04.179147 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:04.179574 26685 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:04.185009 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:04.194126 25100 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:04.197356 24970 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:04.205308 26164 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:04.213641 26555 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:04.249060 28621 raft_consensus.cc:979] T a141985c93da484494d5f82c478d0bc3 P 308436a6f7c7456b99971c666d3bd049: : Instructing follower 6c704dfc19814db996a1c0e363e760ef to start an election
I20240503 02:12:04.249262 28621 raft_consensus.cc:1067] T a141985c93da484494d5f82c478d0bc3 P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Signalling peer 6c704dfc19814db996a1c0e363e760ef to start an election
I20240503 02:12:04.249929 26575 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a141985c93da484494d5f82c478d0bc3"
dest_uuid: "6c704dfc19814db996a1c0e363e760ef"
from {username='slave'} at 127.22.108.1:45997
I20240503 02:12:04.250178 26575 raft_consensus.cc:482] T a141985c93da484494d5f82c478d0bc3 P 6c704dfc19814db996a1c0e363e760ef [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:12:04.250309 26575 raft_consensus.cc:3037] T a141985c93da484494d5f82c478d0bc3 P 6c704dfc19814db996a1c0e363e760ef [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:04.253168 26575 raft_consensus.cc:504] T a141985c93da484494d5f82c478d0bc3 P 6c704dfc19814db996a1c0e363e760ef [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } }
I20240503 02:12:04.254519 26575 leader_election.cc:288] T a141985c93da484494d5f82c478d0bc3 P 6c704dfc19814db996a1c0e363e760ef [CANDIDATE]: Term 2 election: Requested vote from peers 308436a6f7c7456b99971c666d3bd049 (127.22.108.1:40941)
I20240503 02:12:04.254593 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a141985c93da484494d5f82c478d0bc3" candidate_uuid: "6c704dfc19814db996a1c0e363e760ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "308436a6f7c7456b99971c666d3bd049"
I20240503 02:12:04.254804 24990 raft_consensus.cc:3032] T a141985c93da484494d5f82c478d0bc3 P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:12:04.254897 24990 raft_consensus.cc:729] T a141985c93da484494d5f82c478d0bc3 P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 308436a6f7c7456b99971c666d3bd049, State: Running, Role: LEADER
I20240503 02:12:04.255148 24990 consensus_queue.cc:260] T a141985c93da484494d5f82c478d0bc3 P 308436a6f7c7456b99971c666d3bd049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } }
I20240503 02:12:04.255496 24990 raft_consensus.cc:3037] T a141985c93da484494d5f82c478d0bc3 P 308436a6f7c7456b99971c666d3bd049 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:04.258334 24990 raft_consensus.cc:2454] T a141985c93da484494d5f82c478d0bc3 P 308436a6f7c7456b99971c666d3bd049 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c704dfc19814db996a1c0e363e760ef in term 2.
I20240503 02:12:04.258926 26510 leader_election.cc:302] T a141985c93da484494d5f82c478d0bc3 P 6c704dfc19814db996a1c0e363e760ef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 308436a6f7c7456b99971c666d3bd049, 6c704dfc19814db996a1c0e363e760ef; no voters:
I20240503 02:12:04.259524 28620 raft_consensus.cc:2781] T a141985c93da484494d5f82c478d0bc3 P 6c704dfc19814db996a1c0e363e760ef [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:12:04.274008 28620 raft_consensus.cc:686] T a141985c93da484494d5f82c478d0bc3 P 6c704dfc19814db996a1c0e363e760ef [term 2 LEADER]: Becoming Leader. State: Replica: 6c704dfc19814db996a1c0e363e760ef, State: Running, Role: LEADER
I20240503 02:12:04.274559 28620 consensus_queue.cc:237] T a141985c93da484494d5f82c478d0bc3 P 6c704dfc19814db996a1c0e363e760ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } }
I20240503 02:12:04.277776 24840 catalog_manager.cc:5484] T a141985c93da484494d5f82c478d0bc3 P 6c704dfc19814db996a1c0e363e760ef reported cstate change: term changed from 1 to 2, leader changed from 308436a6f7c7456b99971c666d3bd049 (127.22.108.1) to 6c704dfc19814db996a1c0e363e760ef (127.22.108.6). New cstate: current_term: 2 leader_uuid: "6c704dfc19814db996a1c0e363e760ef" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } } }
I20240503 02:12:04.607303 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:04.613582 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:04.616916 26685 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:04.616941 25100 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:04.631518 26164 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:04.638418 26555 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:04.639590 24970 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:04.717846 24990 raft_consensus.cc:1261] T a141985c93da484494d5f82c478d0bc3 P 308436a6f7c7456b99971c666d3bd049 [term 2 FOLLOWER]: Refusing update from remote peer 6c704dfc19814db996a1c0e363e760ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:12:04.718423 28620 consensus_queue.cc:1035] T a141985c93da484494d5f82c478d0bc3 P 6c704dfc19814db996a1c0e363e760ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:12:05.026398 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:05.033527 26685 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:05.047499 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:05.048902 25100 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:05.052129 26555 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:05.060786 26164 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:05.077232 24970 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:05.174703 26575 consensus_queue.cc:237] T a141985c93da484494d5f82c478d0bc3 P 6c704dfc19814db996a1c0e363e760ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } }
I20240503 02:12:05.175513 28708 raft_consensus.cc:2932] T a141985c93da484494d5f82c478d0bc3 P 6c704dfc19814db996a1c0e363e760ef [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 308436a6f7c7456b99971c666d3bd049 (127.22.108.1) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } } }
I20240503 02:12:05.180150 24843 catalog_manager.cc:5484] T a141985c93da484494d5f82c478d0bc3 P 6c704dfc19814db996a1c0e363e760ef reported cstate change: config changed from index 4 to 6, VOTER 308436a6f7c7456b99971c666d3bd049 (127.22.108.1) evicted. New cstate: current_term: 2 leader_uuid: "6c704dfc19814db996a1c0e363e760ef" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } } }
I20240503 02:12:05.184345 24970 tablet_service.cc:1514] Processing DeleteTablet for tablet a141985c93da484494d5f82c478d0bc3 with delete_type TABLET_DATA_TOMBSTONED (TS 308436a6f7c7456b99971c666d3bd049 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47868
I20240503 02:12:05.185001 28812 tablet_replica.cc:330] T a141985c93da484494d5f82c478d0bc3 P 308436a6f7c7456b99971c666d3bd049: stopping tablet replica
I20240503 02:12:05.185228 28812 raft_consensus.cc:2229] T a141985c93da484494d5f82c478d0bc3 P 308436a6f7c7456b99971c666d3bd049 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:05.185451 28812 raft_consensus.cc:2258] T a141985c93da484494d5f82c478d0bc3 P 308436a6f7c7456b99971c666d3bd049 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:05.186206 28812 ts_tablet_manager.cc:1875] T a141985c93da484494d5f82c478d0bc3 P 308436a6f7c7456b99971c666d3bd049: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:05.190945 28812 ts_tablet_manager.cc:1888] T a141985c93da484494d5f82c478d0bc3 P 308436a6f7c7456b99971c666d3bd049: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 02:12:05.191051 28812 log.cc:1196] T a141985c93da484494d5f82c478d0bc3 P 308436a6f7c7456b99971c666d3bd049: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/a141985c93da484494d5f82c478d0bc3
I20240503 02:12:05.191725 24835 catalog_manager.cc:4830] TS 308436a6f7c7456b99971c666d3bd049 (127.22.108.1:40941): tablet a141985c93da484494d5f82c478d0bc3 (table rebalance_test_table_0 [id=adca86e19a68413aa3c2e0c525d683a5]) successfully deleted
I20240503 02:12:05.490556 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:05.493522 26685 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:05.501813 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:05.503319 25100 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:05.514076 24970 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:05.523412 26164 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:05.542652 26555 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:05.708725 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:05.710455 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:05.719954 26685 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:05.722190 25100 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:05.723537 24970 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:05.727524 26555 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:05.744374 26164 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:06.520045 26685 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:06.537523 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:06.561110 25100 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:06.570621 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:06.575811 26555 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:06.597409 26164 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:06.603358 24970 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:06.817698 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:06.821076 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:06.822146 25100 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:06.840224 26685 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:06.840880 26555 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:06.852999 24970 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:06.901434 26164 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:07.204706 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:07.204854 26685 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:07.226413 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:07.226855 25100 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:07.237547 24970 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:07.241343 26555 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:07.275019 26164 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
5a79f2a86741450aa9992d60c94daf6f | 127.22.108.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.108.20:37329 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
0163beb78bfd4d399a70ebacb8f9aa9c | 127.22.108.4:45401 | HEALTHY | <none> | 2 | 0
03afb03fe005462eb07ada37fe23c3cf | 127.22.108.7:34851 | HEALTHY | <none> | 2 | 0
1b2a71b48d4e40679504230af08279e9 | 127.22.108.5:40741 | HEALTHY | <none> | 2 | 0
22b0b1ae6c504301ad5170c6bdcd179e | 127.22.108.2:36213 | HEALTHY | <none> | 2 | 0
308436a6f7c7456b99971c666d3bd049 | 127.22.108.1:40941 | HEALTHY | <none> | 3 | 0
6c704dfc19814db996a1c0e363e760ef | 127.22.108.6:43731 | HEALTHY | <none> | 3 | 0
b95ea99cb72f49c6b42c7e8e293cfc27 | 127.22.108.3:36823 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.22.108.1 | experimental | 127.22.108.1:40941
local_ip_for_outbound_sockets | 127.22.108.2 | experimental | 127.22.108.2:36213
local_ip_for_outbound_sockets | 127.22.108.3 | experimental | 127.22.108.3:36823
local_ip_for_outbound_sockets | 127.22.108.4 | experimental | 127.22.108.4:45401
local_ip_for_outbound_sockets | 127.22.108.5 | experimental | 127.22.108.5:40741
local_ip_for_outbound_sockets | 127.22.108.6 | experimental | 127.22.108.6:43731
local_ip_for_outbound_sockets | 127.22.108.7 | experimental | 127.22.108.7:34851
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.22.108.1:40941
server_dump_info_path | /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.108.2:36213
server_dump_info_path | /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.108.3:36823
server_dump_info_path | /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.22.108.4:45401
server_dump_info_path | /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.108.5:40741
server_dump_info_path | /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.108.6:43731
server_dump_info_path | /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.22.108.7:34851
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.108.20:37329 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 02:12:07.324252 22960 log_verifier.cc:126] Checking tablet 4ce4d66aff0e48deb5513001f1d81963
I20240503 02:12:07.340685 22960 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 4ce4d66aff0e48deb5513001f1d81963
I20240503 02:12:07.340858 22960 log_verifier.cc:126] Checking tablet 4e5d560763774c0cba004753269a158c
I20240503 02:12:07.353523 22960 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 4e5d560763774c0cba004753269a158c
I20240503 02:12:07.353658 22960 log_verifier.cc:126] Checking tablet 742a25555073462faaabd4af58f42017
I20240503 02:12:07.366683 22960 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 742a25555073462faaabd4af58f42017
I20240503 02:12:07.366816 22960 log_verifier.cc:126] Checking tablet 7b671abb263d4848b7b909927b7df380
I20240503 02:12:07.382711 22960 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 7b671abb263d4848b7b909927b7df380
I20240503 02:12:07.382845 22960 log_verifier.cc:126] Checking tablet 7c0b16db9b77442b922e3b7bf1247878
I20240503 02:12:07.395607 22960 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 7c0b16db9b77442b922e3b7bf1247878
I20240503 02:12:07.395735 22960 log_verifier.cc:126] Checking tablet 88000b61106c4087948a9aa37ecc297f
I20240503 02:12:07.408030 22960 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 88000b61106c4087948a9aa37ecc297f
I20240503 02:12:07.408152 22960 log_verifier.cc:126] Checking tablet 923672905ffb452993c5dcf283f4a399
I20240503 02:12:07.420192 22960 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 923672905ffb452993c5dcf283f4a399
I20240503 02:12:07.420323 22960 log_verifier.cc:126] Checking tablet 92f95fea174e43c9bc3aa019d2191e51
I20240503 02:12:07.432077 22960 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 92f95fea174e43c9bc3aa019d2191e51
I20240503 02:12:07.432205 22960 log_verifier.cc:126] Checking tablet a141985c93da484494d5f82c478d0bc3
I20240503 02:12:07.444995 22960 log_verifier.cc:177] Verified matching terms for 6 ops in tablet a141985c93da484494d5f82c478d0bc3
I20240503 02:12:07.445169 22960 log_verifier.cc:126] Checking tablet af70ad6a7a8a4051beb408ead22d30e3
I20240503 02:12:07.456938 22960 log_verifier.cc:177] Verified matching terms for 3 ops in tablet af70ad6a7a8a4051beb408ead22d30e3
I20240503 02:12:07.457098 22960 log_verifier.cc:126] Checking tablet bb1a9efdfd96421ea17c41cfa8977f36
I20240503 02:12:07.469626 22960 log_verifier.cc:177] Verified matching terms for 6 ops in tablet bb1a9efdfd96421ea17c41cfa8977f36
I20240503 02:12:07.469756 22960 log_verifier.cc:126] Checking tablet e1093d22a1dc4e0f81fd3df39bf03265
I20240503 02:12:07.482118 22960 log_verifier.cc:177] Verified matching terms for 6 ops in tablet e1093d22a1dc4e0f81fd3df39bf03265
I20240503 02:12:07.482249 22960 log_verifier.cc:126] Checking tablet ed43f867d21d4bee9efc0b9bf4ee90c1
I20240503 02:12:07.494378 22960 log_verifier.cc:177] Verified matching terms for 6 ops in tablet ed43f867d21d4bee9efc0b9bf4ee90c1
I20240503 02:12:07.494519 22960 log_verifier.cc:126] Checking tablet ee1a68079bfa435ab820967fb894d3f1
I20240503 02:12:07.502763 22960 log_verifier.cc:177] Verified matching terms for 4 ops in tablet ee1a68079bfa435ab820967fb894d3f1
I20240503 02:12:07.502899 22960 log_verifier.cc:126] Checking tablet f6920bfd97624ad381fd72b688ab73f1
I20240503 02:12:07.515647 22960 log_verifier.cc:177] Verified matching terms for 6 ops in tablet f6920bfd97624ad381fd72b688ab73f1
I20240503 02:12:07.515775 22960 log_verifier.cc:126] Checking tablet fb78606eb2e74e19ab30bf7a119ea7e6
I20240503 02:12:07.527985 22960 log_verifier.cc:177] Verified matching terms for 3 ops in tablet fb78606eb2e74e19ab30bf7a119ea7e6
I20240503 02:12:07.531093 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.8:0
--local_ip_for_outbound_sockets=127.22.108.8
--webserver_interface=127.22.108.8
--webserver_port=0
--tserver_master_addrs=127.22.108.62:38409
--builtin_ntp_servers=127.22.108.20:37329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 02:12:07.880759 28996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:07.881124 28996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:07.881273 28996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:07.889194 28996 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:12:07.889307 28996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:07.889470 28996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.8
I20240503 02:12:07.899181 28996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:37329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.22.108.8
--webserver_port=0
--tserver_master_addrs=127.22.108.62:38409
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.8
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:07.900532 28996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:07.902508 28996 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:07.915366 29002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:07.916023 29004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:07.916646 29001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:07.917559 28996 server_base.cc:1034] running on GCE node
I20240503 02:12:07.918172 28996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:07.919165 28996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:07.920408 28996 hybrid_clock.cc:648] HybridClock initialized: now 1714702327920324 us; error 56 us; skew 500 ppm
I20240503 02:12:07.920766 28996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:07.923146 28996 webserver.cc:460] Webserver started at http://127.22.108.8:38997/ using document root <none> and password file <none>
I20240503 02:12:07.923911 28996 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:07.924008 28996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:07.924270 28996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:07.927349 28996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "20b9275d1c1a4f6c81b8555d956cd7bc"
format_stamp: "Formatted at 2024-05-03 02:12:07 on dist-test-slave-sc15"
I20240503 02:12:07.928161 28996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "20b9275d1c1a4f6c81b8555d956cd7bc"
format_stamp: "Formatted at 2024-05-03 02:12:07 on dist-test-slave-sc15"
I20240503 02:12:07.933626 28996 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240503 02:12:07.937302 29010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:07.938735 28996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:12:07.938907 28996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "20b9275d1c1a4f6c81b8555d956cd7bc"
format_stamp: "Formatted at 2024-05-03 02:12:07 on dist-test-slave-sc15"
I20240503 02:12:07.939121 28996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:07.956058 28996 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:07.956954 28996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:07.957269 28996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:07.958261 28996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:07.959769 28996 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:07.959870 28996 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:07.959995 28996 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:07.960067 28996 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:07.997206 28996 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.8:43455
I20240503 02:12:07.997247 29122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.8:43455 every 8 connection(s)
I20240503 02:12:07.998654 28996 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 02:12:08.003007 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 28996
I20240503 02:12:08.003242 22960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 02:12:08.013391 29123 heartbeater.cc:347] Connected to a master server at 127.22.108.62:38409
I20240503 02:12:08.013769 29123 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:08.014779 29123 heartbeater.cc:511] Master 127.22.108.62:38409 requested a full tablet report, sending...
I20240503 02:12:08.016521 24846 ts_manager.cc:194] Registered new tserver with Master: 20b9275d1c1a4f6c81b8555d956cd7bc (127.22.108.8:43455)
I20240503 02:12:08.017380 24846 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.8:43859
I20240503 02:12:08.548475 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:08.566025 26685 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:08.566538 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:08.570191 24970 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:08.580632 26555 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:08.617312 25100 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:08.642750 29058 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:08.644656 26164 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:08.851208 29058 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:08.854638 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:08.864385 26685 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:08.871837 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:08.881583 24970 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:08.904104 25100 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:08.917129 26555 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:08.918141 26164 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:09.020637 29123 heartbeater.cc:503] Master 127.22.108.62:38409 was elected leader, sending a full tablet report...
I20240503 02:12:09.031765 26575 consensus_queue.cc:237] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } } peers { permanent_uuid: "20b9275d1c1a4f6c81b8555d956cd7bc" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 43455 } }
W20240503 02:12:09.041304 26509 consensus_peers.cc:485] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef -> Peer 20b9275d1c1a4f6c81b8555d956cd7bc (127.22.108.8:43455): Couldn't send request to peer 20b9275d1c1a4f6c81b8555d956cd7bc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e5d560763774c0cba004753269a158c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:09.430497 29229 ts_tablet_manager.cc:902] T 4e5d560763774c0cba004753269a158c P 20b9275d1c1a4f6c81b8555d956cd7bc: Initiating tablet copy from peer 6c704dfc19814db996a1c0e363e760ef (127.22.108.6:43731)
I20240503 02:12:09.432111 29229 tablet_copy_client.cc:308] T 4e5d560763774c0cba004753269a158c P 20b9275d1c1a4f6c81b8555d956cd7bc: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.6:43731
I20240503 02:12:09.439033 26595 tablet_copy_service.cc:140] P 6c704dfc19814db996a1c0e363e760ef: Received BeginTabletCopySession request for tablet 4e5d560763774c0cba004753269a158c from peer 20b9275d1c1a4f6c81b8555d956cd7bc ({username='slave'} at 127.22.108.8:59141)
I20240503 02:12:09.439263 26595 tablet_copy_service.cc:161] P 6c704dfc19814db996a1c0e363e760ef: Beginning new tablet copy session on tablet 4e5d560763774c0cba004753269a158c from peer 20b9275d1c1a4f6c81b8555d956cd7bc at {username='slave'} at 127.22.108.8:59141: session id = 20b9275d1c1a4f6c81b8555d956cd7bc-4e5d560763774c0cba004753269a158c
I20240503 02:12:09.441793 26595 tablet_copy_source_session.cc:205] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:09.446517 29229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e5d560763774c0cba004753269a158c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:09.455492 29229 tablet_copy_client.cc:783] T 4e5d560763774c0cba004753269a158c P 20b9275d1c1a4f6c81b8555d956cd7bc: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:09.455864 29229 tablet_copy_client.cc:647] T 4e5d560763774c0cba004753269a158c P 20b9275d1c1a4f6c81b8555d956cd7bc: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:09.458545 29229 tablet_copy_client.cc:523] T 4e5d560763774c0cba004753269a158c P 20b9275d1c1a4f6c81b8555d956cd7bc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:09.461778 29229 tablet_bootstrap.cc:492] T 4e5d560763774c0cba004753269a158c P 20b9275d1c1a4f6c81b8555d956cd7bc: Bootstrap starting.
I20240503 02:12:09.471493 29229 log.cc:824] T 4e5d560763774c0cba004753269a158c P 20b9275d1c1a4f6c81b8555d956cd7bc: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:09.502586 29229 tablet_bootstrap.cc:492] T 4e5d560763774c0cba004753269a158c P 20b9275d1c1a4f6c81b8555d956cd7bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 02:12:09.503162 29229 tablet_bootstrap.cc:492] T 4e5d560763774c0cba004753269a158c P 20b9275d1c1a4f6c81b8555d956cd7bc: Bootstrap complete.
I20240503 02:12:09.503598 29229 ts_tablet_manager.cc:1367] T 4e5d560763774c0cba004753269a158c P 20b9275d1c1a4f6c81b8555d956cd7bc: Time spent bootstrapping tablet: real 0.042s user 0.036s sys 0.003s
I20240503 02:12:09.509143 29229 raft_consensus.cc:348] T 4e5d560763774c0cba004753269a158c P 20b9275d1c1a4f6c81b8555d956cd7bc [term 2 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } }
I20240503 02:12:09.510062 29229 raft_consensus.cc:729] T 4e5d560763774c0cba004753269a158c P 20b9275d1c1a4f6c81b8555d956cd7bc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b9275d1c1a4f6c81b8555d956cd7bc, State: Initialized, Role: FOLLOWER
I20240503 02:12:09.510484 29229 consensus_queue.cc:260] T 4e5d560763774c0cba004753269a158c P 20b9275d1c1a4f6c81b8555d956cd7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 2.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } } peers { permanent_uuid: "20b9275d1c1a4f6c81b8555d956cd7bc" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 43455 } }
I20240503 02:12:09.511883 29229 ts_tablet_manager.cc:1398] T 4e5d560763774c0cba004753269a158c P 20b9275d1c1a4f6c81b8555d956cd7bc: Time spent starting tablet: real 0.008s user 0.006s sys 0.004s
I20240503 02:12:09.512881 26595 tablet_copy_service.cc:342] P 6c704dfc19814db996a1c0e363e760ef: Request end of tablet copy session 20b9275d1c1a4f6c81b8555d956cd7bc-4e5d560763774c0cba004753269a158c received from {username='slave'} at 127.22.108.8:59141
I20240503 02:12:09.513118 26595 tablet_copy_service.cc:433] P 6c704dfc19814db996a1c0e363e760ef: ending tablet copy session 20b9275d1c1a4f6c81b8555d956cd7bc-4e5d560763774c0cba004753269a158c on tablet 4e5d560763774c0cba004753269a158c with peer 20b9275d1c1a4f6c81b8555d956cd7bc
I20240503 02:12:10.040185 29078 raft_consensus.cc:1203] T 4e5d560763774c0cba004753269a158c P 20b9275d1c1a4f6c81b8555d956cd7bc [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.6->[2.7-2.7] Dedup: 2.7->[]
I20240503 02:12:10.041807 29239 raft_consensus.cc:2932] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 20b9275d1c1a4f6c81b8555d956cd7bc (127.22.108.8) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } } peers { permanent_uuid: "20b9275d1c1a4f6c81b8555d956cd7bc" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 43455 } } }
I20240503 02:12:10.042548 29078 raft_consensus.cc:2932] T 4e5d560763774c0cba004753269a158c P 20b9275d1c1a4f6c81b8555d956cd7bc [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 20b9275d1c1a4f6c81b8555d956cd7bc (127.22.108.8) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } } peers { permanent_uuid: "20b9275d1c1a4f6c81b8555d956cd7bc" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 43455 } } }
I20240503 02:12:10.044816 29239 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:43856 (request call id 0) took 1013 ms. Trace:
I20240503 02:12:10.045049 29239 rpcz_store.cc:276] 0503 02:12:09.031141 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:09.031211 (+ 70us) service_pool.cc:225] Handling call
0503 02:12:09.032476 (+ 1265us) log.cc:842] Serialized 248 byte log entry
0503 02:12:10.044763 (+1012287us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":87,"raft.queue_time_us":433,"raft.run_cpu_time_us":301,"raft.run_wall_time_us":301,"spinlock_wait_cycles":105344,"thread_start_us":546,"threads_started":2,"wal-append.queue_time_us":312,"wal-append.run_cpu_time_us":562,"wal-append.run_wall_time_us":1000721}
I20240503 02:12:10.048090 24846 catalog_manager.cc:5484] T 4e5d560763774c0cba004753269a158c P 20b9275d1c1a4f6c81b8555d956cd7bc reported cstate change: config changed from index 6 to 7, VOTER 20b9275d1c1a4f6c81b8555d956cd7bc (127.22.108.8) added. New cstate: current_term: 2 leader_uuid: "6c704dfc19814db996a1c0e363e760ef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } } peers { permanent_uuid: "20b9275d1c1a4f6c81b8555d956cd7bc" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 43455 } } }
I20240503 02:12:10.081929 24990 consensus_queue.cc:237] T 88000b61106c4087948a9aa37ecc297f P 308436a6f7c7456b99971c666d3bd049 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "20b9275d1c1a4f6c81b8555d956cd7bc" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 43455 } }
W20240503 02:12:10.090392 24924 consensus_peers.cc:485] T 88000b61106c4087948a9aa37ecc297f P 308436a6f7c7456b99971c666d3bd049 -> Peer 20b9275d1c1a4f6c81b8555d956cd7bc (127.22.108.8:43455): Couldn't send request to peer 20b9275d1c1a4f6c81b8555d956cd7bc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 88000b61106c4087948a9aa37ecc297f. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:10.571370 29267 ts_tablet_manager.cc:902] T 88000b61106c4087948a9aa37ecc297f P 20b9275d1c1a4f6c81b8555d956cd7bc: Initiating tablet copy from peer 308436a6f7c7456b99971c666d3bd049 (127.22.108.1:40941)
I20240503 02:12:10.572551 29267 tablet_copy_client.cc:308] T 88000b61106c4087948a9aa37ecc297f P 20b9275d1c1a4f6c81b8555d956cd7bc: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.1:40941
I20240503 02:12:10.578198 25010 tablet_copy_service.cc:140] P 308436a6f7c7456b99971c666d3bd049: Received BeginTabletCopySession request for tablet 88000b61106c4087948a9aa37ecc297f from peer 20b9275d1c1a4f6c81b8555d956cd7bc ({username='slave'} at 127.22.108.8:49711)
I20240503 02:12:10.578353 25010 tablet_copy_service.cc:161] P 308436a6f7c7456b99971c666d3bd049: Beginning new tablet copy session on tablet 88000b61106c4087948a9aa37ecc297f from peer 20b9275d1c1a4f6c81b8555d956cd7bc at {username='slave'} at 127.22.108.8:49711: session id = 20b9275d1c1a4f6c81b8555d956cd7bc-88000b61106c4087948a9aa37ecc297f
I20240503 02:12:10.580495 25010 tablet_copy_source_session.cc:205] T 88000b61106c4087948a9aa37ecc297f P 308436a6f7c7456b99971c666d3bd049: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:10.581840 29267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88000b61106c4087948a9aa37ecc297f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:10.586040 29267 tablet_copy_client.cc:783] T 88000b61106c4087948a9aa37ecc297f P 20b9275d1c1a4f6c81b8555d956cd7bc: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:10.586261 29267 tablet_copy_client.cc:647] T 88000b61106c4087948a9aa37ecc297f P 20b9275d1c1a4f6c81b8555d956cd7bc: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:10.588223 29267 tablet_copy_client.cc:523] T 88000b61106c4087948a9aa37ecc297f P 20b9275d1c1a4f6c81b8555d956cd7bc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:10.590785 29267 tablet_bootstrap.cc:492] T 88000b61106c4087948a9aa37ecc297f P 20b9275d1c1a4f6c81b8555d956cd7bc: Bootstrap starting.
I20240503 02:12:10.621268 29267 tablet_bootstrap.cc:492] T 88000b61106c4087948a9aa37ecc297f P 20b9275d1c1a4f6c81b8555d956cd7bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 02:12:10.621770 29267 tablet_bootstrap.cc:492] T 88000b61106c4087948a9aa37ecc297f P 20b9275d1c1a4f6c81b8555d956cd7bc: Bootstrap complete.
I20240503 02:12:10.622089 29267 ts_tablet_manager.cc:1367] T 88000b61106c4087948a9aa37ecc297f P 20b9275d1c1a4f6c81b8555d956cd7bc: Time spent bootstrapping tablet: real 0.031s user 0.033s sys 0.000s
I20240503 02:12:10.623102 29267 raft_consensus.cc:348] T 88000b61106c4087948a9aa37ecc297f P 20b9275d1c1a4f6c81b8555d956cd7bc [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } }
I20240503 02:12:10.623507 29267 raft_consensus.cc:729] T 88000b61106c4087948a9aa37ecc297f P 20b9275d1c1a4f6c81b8555d956cd7bc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b9275d1c1a4f6c81b8555d956cd7bc, State: Initialized, Role: FOLLOWER
I20240503 02:12:10.623706 29267 consensus_queue.cc:260] T 88000b61106c4087948a9aa37ecc297f P 20b9275d1c1a4f6c81b8555d956cd7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "20b9275d1c1a4f6c81b8555d956cd7bc" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 43455 } }
I20240503 02:12:10.624477 29267 ts_tablet_manager.cc:1398] T 88000b61106c4087948a9aa37ecc297f P 20b9275d1c1a4f6c81b8555d956cd7bc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:10.625375 25010 tablet_copy_service.cc:342] P 308436a6f7c7456b99971c666d3bd049: Request end of tablet copy session 20b9275d1c1a4f6c81b8555d956cd7bc-88000b61106c4087948a9aa37ecc297f received from {username='slave'} at 127.22.108.8:49711
I20240503 02:12:10.625558 25010 tablet_copy_service.cc:433] P 308436a6f7c7456b99971c666d3bd049: ending tablet copy session 20b9275d1c1a4f6c81b8555d956cd7bc-88000b61106c4087948a9aa37ecc297f on tablet 88000b61106c4087948a9aa37ecc297f with peer 20b9275d1c1a4f6c81b8555d956cd7bc
I20240503 02:12:11.148167 29078 raft_consensus.cc:1203] T 88000b61106c4087948a9aa37ecc297f P 20b9275d1c1a4f6c81b8555d956cd7bc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:12:11.149905 29278 raft_consensus.cc:2932] T 88000b61106c4087948a9aa37ecc297f P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 20b9275d1c1a4f6c81b8555d956cd7bc (127.22.108.8) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "20b9275d1c1a4f6c81b8555d956cd7bc" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 43455 } } }
I20240503 02:12:11.150537 29078 raft_consensus.cc:2932] T 88000b61106c4087948a9aa37ecc297f P 20b9275d1c1a4f6c81b8555d956cd7bc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 20b9275d1c1a4f6c81b8555d956cd7bc (127.22.108.8) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "20b9275d1c1a4f6c81b8555d956cd7bc" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 43455 } } }
I20240503 02:12:11.153259 29278 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:40066 (request call id 0) took 1072 ms. Trace:
I20240503 02:12:11.153476 29278 rpcz_store.cc:276] 0503 02:12:10.081226 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:10.081314 (+ 88us) service_pool.cc:225] Handling call
0503 02:12:10.082643 (+ 1329us) log.cc:842] Serialized 257 byte log entry
0503 02:12:11.153209 (+1070566us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":76,"raft.queue_time_us":346,"raft.run_cpu_time_us":311,"raft.run_wall_time_us":312,"thread_start_us":471,"threads_started":2,"wal-append.queue_time_us":317,"wal-append.run_cpu_time_us":612,"wal-append.run_wall_time_us":1000674}
I20240503 02:12:11.155215 24845 catalog_manager.cc:5484] T 88000b61106c4087948a9aa37ecc297f P 20b9275d1c1a4f6c81b8555d956cd7bc reported cstate change: config changed from index -1 to 4, VOTER 20b9275d1c1a4f6c81b8555d956cd7bc (127.22.108.8) added. New cstate: current_term: 1 leader_uuid: "308436a6f7c7456b99971c666d3bd049" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "20b9275d1c1a4f6c81b8555d956cd7bc" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 43455 } } }
I20240503 02:12:11.281381 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:11.283923 26685 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:11.286020 24970 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:11.287185 26555 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:11.311426 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:11.331768 29058 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:11.332540 25100 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:11.353047 26164 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:11.406946 24990 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "88000b61106c4087948a9aa37ecc297f"
dest_uuid: "308436a6f7c7456b99971c666d3bd049"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40092
I20240503 02:12:11.407182 24990 raft_consensus.cc:595] T 88000b61106c4087948a9aa37ecc297f P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:12:11.496526 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:11.503870 26685 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:11.524868 24970 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:11.526769 29278 raft_consensus.cc:979] T 88000b61106c4087948a9aa37ecc297f P 308436a6f7c7456b99971c666d3bd049: : Instructing follower 20b9275d1c1a4f6c81b8555d956cd7bc to start an election
I20240503 02:12:11.526938 29278 raft_consensus.cc:1067] T 88000b61106c4087948a9aa37ecc297f P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Signalling peer 20b9275d1c1a4f6c81b8555d956cd7bc to start an election
I20240503 02:12:11.527204 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:11.527386 25100 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:11.528246 29078 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "88000b61106c4087948a9aa37ecc297f"
dest_uuid: "20b9275d1c1a4f6c81b8555d956cd7bc"
from {username='slave'} at 127.22.108.1:60621
I20240503 02:12:11.528554 29078 raft_consensus.cc:482] T 88000b61106c4087948a9aa37ecc297f P 20b9275d1c1a4f6c81b8555d956cd7bc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:12:11.528782 29078 raft_consensus.cc:3037] T 88000b61106c4087948a9aa37ecc297f P 20b9275d1c1a4f6c81b8555d956cd7bc [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:11.531721 29078 raft_consensus.cc:504] T 88000b61106c4087948a9aa37ecc297f P 20b9275d1c1a4f6c81b8555d956cd7bc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "20b9275d1c1a4f6c81b8555d956cd7bc" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 43455 } }
I20240503 02:12:11.533753 29078 leader_election.cc:288] T 88000b61106c4087948a9aa37ecc297f P 20b9275d1c1a4f6c81b8555d956cd7bc [CANDIDATE]: Term 2 election: Requested vote from peers 308436a6f7c7456b99971c666d3bd049 (127.22.108.1:40941)
I20240503 02:12:11.534024 24990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88000b61106c4087948a9aa37ecc297f" candidate_uuid: "20b9275d1c1a4f6c81b8555d956cd7bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "308436a6f7c7456b99971c666d3bd049"
I20240503 02:12:11.534268 24990 raft_consensus.cc:3032] T 88000b61106c4087948a9aa37ecc297f P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:12:11.534373 24990 raft_consensus.cc:729] T 88000b61106c4087948a9aa37ecc297f P 308436a6f7c7456b99971c666d3bd049 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 308436a6f7c7456b99971c666d3bd049, State: Running, Role: LEADER
I20240503 02:12:11.534643 24990 consensus_queue.cc:260] T 88000b61106c4087948a9aa37ecc297f P 308436a6f7c7456b99971c666d3bd049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "20b9275d1c1a4f6c81b8555d956cd7bc" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 43455 } }
I20240503 02:12:11.534955 24990 raft_consensus.cc:3037] T 88000b61106c4087948a9aa37ecc297f P 308436a6f7c7456b99971c666d3bd049 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:11.538455 24990 raft_consensus.cc:2454] T 88000b61106c4087948a9aa37ecc297f P 308436a6f7c7456b99971c666d3bd049 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b9275d1c1a4f6c81b8555d956cd7bc in term 2.
I20240503 02:12:11.539780 26555 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:11.541657 29013 leader_election.cc:302] T 88000b61106c4087948a9aa37ecc297f P 20b9275d1c1a4f6c81b8555d956cd7bc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 20b9275d1c1a4f6c81b8555d956cd7bc, 308436a6f7c7456b99971c666d3bd049; no voters:
I20240503 02:12:11.542340 29277 raft_consensus.cc:2781] T 88000b61106c4087948a9aa37ecc297f P 20b9275d1c1a4f6c81b8555d956cd7bc [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:12:11.542816 29277 raft_consensus.cc:686] T 88000b61106c4087948a9aa37ecc297f P 20b9275d1c1a4f6c81b8555d956cd7bc [term 2 LEADER]: Becoming Leader. State: Replica: 20b9275d1c1a4f6c81b8555d956cd7bc, State: Running, Role: LEADER
I20240503 02:12:11.543373 29277 consensus_queue.cc:237] T 88000b61106c4087948a9aa37ecc297f P 20b9275d1c1a4f6c81b8555d956cd7bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "20b9275d1c1a4f6c81b8555d956cd7bc" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 43455 } }
I20240503 02:12:11.549501 29058 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:11.558152 26164 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:11.558045 24843 catalog_manager.cc:5484] T 88000b61106c4087948a9aa37ecc297f P 20b9275d1c1a4f6c81b8555d956cd7bc reported cstate change: term changed from 1 to 2, leader changed from 308436a6f7c7456b99971c666d3bd049 (127.22.108.1) to 20b9275d1c1a4f6c81b8555d956cd7bc (127.22.108.8). New cstate: current_term: 2 leader_uuid: "20b9275d1c1a4f6c81b8555d956cd7bc" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 } } peers { permanent_uuid: "20b9275d1c1a4f6c81b8555d956cd7bc" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 43455 } } }
I20240503 02:12:11.748580 26685 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:11.748946 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:11.750516 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:11.757371 29058 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:11.768694 24970 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:11.783025 25100 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:11.790588 26164 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:11.807986 26555 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:11.875586 26575 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4e5d560763774c0cba004753269a158c"
dest_uuid: "6c704dfc19814db996a1c0e363e760ef"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43902
I20240503 02:12:11.875911 26575 raft_consensus.cc:595] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef [term 2 LEADER]: Received request to transfer leadership
I20240503 02:12:11.973438 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:11.981534 29058 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:11.984046 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:11.985195 25100 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:11.996261 26685 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:12.024698 26555 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:12.039836 24970 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:12.039937 26164 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:12.140185 24990 raft_consensus.cc:1261] T 88000b61106c4087948a9aa37ecc297f P 308436a6f7c7456b99971c666d3bd049 [term 2 FOLLOWER]: Refusing update from remote peer 20b9275d1c1a4f6c81b8555d956cd7bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:12:12.140870 29449 consensus_queue.cc:1035] T 88000b61106c4087948a9aa37ecc297f P 20b9275d1c1a4f6c81b8555d956cd7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "308436a6f7c7456b99971c666d3bd049" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 40941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:12:12.220566 29454 raft_consensus.cc:979] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef: : Instructing follower 20b9275d1c1a4f6c81b8555d956cd7bc to start an election
I20240503 02:12:12.220743 29454 raft_consensus.cc:1067] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef [term 2 LEADER]: Signalling peer 20b9275d1c1a4f6c81b8555d956cd7bc to start an election
I20240503 02:12:12.221473 29078 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e5d560763774c0cba004753269a158c"
dest_uuid: "20b9275d1c1a4f6c81b8555d956cd7bc"
from {username='slave'} at 127.22.108.6:50361
I20240503 02:12:12.221683 29078 raft_consensus.cc:482] T 4e5d560763774c0cba004753269a158c P 20b9275d1c1a4f6c81b8555d956cd7bc [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:12:12.221769 29078 raft_consensus.cc:3037] T 4e5d560763774c0cba004753269a158c P 20b9275d1c1a4f6c81b8555d956cd7bc [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:12:12.223584 29078 raft_consensus.cc:504] T 4e5d560763774c0cba004753269a158c P 20b9275d1c1a4f6c81b8555d956cd7bc [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } } peers { permanent_uuid: "20b9275d1c1a4f6c81b8555d956cd7bc" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 43455 } }
I20240503 02:12:12.224244 29078 leader_election.cc:288] T 4e5d560763774c0cba004753269a158c P 20b9275d1c1a4f6c81b8555d956cd7bc [CANDIDATE]: Term 3 election: Requested vote from peers 6c704dfc19814db996a1c0e363e760ef (127.22.108.6:43731)
I20240503 02:12:12.224858 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5d560763774c0cba004753269a158c" candidate_uuid: "20b9275d1c1a4f6c81b8555d956cd7bc" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "6c704dfc19814db996a1c0e363e760ef"
I20240503 02:12:12.225183 26575 raft_consensus.cc:3032] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:12:12.225276 26575 raft_consensus.cc:729] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6c704dfc19814db996a1c0e363e760ef, State: Running, Role: LEADER
I20240503 02:12:12.225564 26575 consensus_queue.cc:260] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } } peers { permanent_uuid: "20b9275d1c1a4f6c81b8555d956cd7bc" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 43455 } }
I20240503 02:12:12.225880 26575 raft_consensus.cc:3037] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:12:12.227996 26575 raft_consensus.cc:2454] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b9275d1c1a4f6c81b8555d956cd7bc in term 3.
I20240503 02:12:12.228464 29013 leader_election.cc:302] T 4e5d560763774c0cba004753269a158c P 20b9275d1c1a4f6c81b8555d956cd7bc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 20b9275d1c1a4f6c81b8555d956cd7bc, 6c704dfc19814db996a1c0e363e760ef; no voters:
I20240503 02:12:12.228794 29451 raft_consensus.cc:2781] T 4e5d560763774c0cba004753269a158c P 20b9275d1c1a4f6c81b8555d956cd7bc [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:12:12.228916 29451 raft_consensus.cc:686] T 4e5d560763774c0cba004753269a158c P 20b9275d1c1a4f6c81b8555d956cd7bc [term 3 LEADER]: Becoming Leader. State: Replica: 20b9275d1c1a4f6c81b8555d956cd7bc, State: Running, Role: LEADER
I20240503 02:12:12.229226 29451 consensus_queue.cc:237] T 4e5d560763774c0cba004753269a158c P 20b9275d1c1a4f6c81b8555d956cd7bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } } peers { permanent_uuid: "20b9275d1c1a4f6c81b8555d956cd7bc" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 43455 } }
I20240503 02:12:12.232262 24848 catalog_manager.cc:5484] T 4e5d560763774c0cba004753269a158c P 20b9275d1c1a4f6c81b8555d956cd7bc reported cstate change: term changed from 2 to 3, leader changed from 6c704dfc19814db996a1c0e363e760ef (127.22.108.6) to 20b9275d1c1a4f6c81b8555d956cd7bc (127.22.108.8). New cstate: current_term: 3 leader_uuid: "20b9275d1c1a4f6c81b8555d956cd7bc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 } } peers { permanent_uuid: "20b9275d1c1a4f6c81b8555d956cd7bc" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 43455 } } }
I20240503 02:12:12.430258 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:12.447405 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:12.451090 29058 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:12.452206 26685 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:12.461529 24970 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:12.472209 26164 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:12.473661 26555 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:12.506316 25100 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:12.581650 29078 consensus_queue.cc:237] T 88000b61106c4087948a9aa37ecc297f P 20b9275d1c1a4f6c81b8555d956cd7bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "20b9275d1c1a4f6c81b8555d956cd7bc" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 43455 } }
I20240503 02:12:12.582402 29451 raft_consensus.cc:2932] T 88000b61106c4087948a9aa37ecc297f P 20b9275d1c1a4f6c81b8555d956cd7bc [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 308436a6f7c7456b99971c666d3bd049 (127.22.108.1) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "20b9275d1c1a4f6c81b8555d956cd7bc" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 43455 } } }
I20240503 02:12:12.586812 24846 catalog_manager.cc:5484] T 88000b61106c4087948a9aa37ecc297f P 20b9275d1c1a4f6c81b8555d956cd7bc reported cstate change: config changed from index 4 to 6, VOTER 308436a6f7c7456b99971c666d3bd049 (127.22.108.1) evicted. New cstate: current_term: 2 leader_uuid: "20b9275d1c1a4f6c81b8555d956cd7bc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "20b9275d1c1a4f6c81b8555d956cd7bc" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 43455 } } }
I20240503 02:12:12.590279 24970 tablet_service.cc:1514] Processing DeleteTablet for tablet 88000b61106c4087948a9aa37ecc297f with delete_type TABLET_DATA_TOMBSTONED (TS 308436a6f7c7456b99971c666d3bd049 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47868
I20240503 02:12:12.590991 29514 tablet_replica.cc:330] T 88000b61106c4087948a9aa37ecc297f P 308436a6f7c7456b99971c666d3bd049: stopping tablet replica
I20240503 02:12:12.591174 29514 raft_consensus.cc:2229] T 88000b61106c4087948a9aa37ecc297f P 308436a6f7c7456b99971c666d3bd049 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:12.591380 29514 raft_consensus.cc:2258] T 88000b61106c4087948a9aa37ecc297f P 308436a6f7c7456b99971c666d3bd049 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:12.593564 29514 ts_tablet_manager.cc:1875] T 88000b61106c4087948a9aa37ecc297f P 308436a6f7c7456b99971c666d3bd049: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:12.598829 29514 ts_tablet_manager.cc:1888] T 88000b61106c4087948a9aa37ecc297f P 308436a6f7c7456b99971c666d3bd049: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 02:12:12.598970 29514 log.cc:1196] T 88000b61106c4087948a9aa37ecc297f P 308436a6f7c7456b99971c666d3bd049: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/88000b61106c4087948a9aa37ecc297f
I20240503 02:12:12.599820 24835 catalog_manager.cc:4830] TS 308436a6f7c7456b99971c666d3bd049 (127.22.108.1:40941): tablet 88000b61106c4087948a9aa37ecc297f (table rebalance_test_table_2 [id=7aa88098eef344a7aebdefe66f69aa8b]) successfully deleted
I20240503 02:12:12.620522 26575 raft_consensus.cc:1261] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef [term 3 FOLLOWER]: Refusing update from remote peer 20b9275d1c1a4f6c81b8555d956cd7bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240503 02:12:12.621325 29451 consensus_queue.cc:1035] T 4e5d560763774c0cba004753269a158c P 20b9275d1c1a4f6c81b8555d956cd7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c704dfc19814db996a1c0e363e760ef" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 43731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:12:12.696070 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:12.703835 26685 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:12.707216 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:12.717852 29058 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:12.721266 24970 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:12.741140 26555 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:12.768580 25100 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:12.780578 26164 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:12.872067 29078 consensus_queue.cc:237] T 4e5d560763774c0cba004753269a158c P 20b9275d1c1a4f6c81b8555d956cd7bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 7, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "20b9275d1c1a4f6c81b8555d956cd7bc" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 43455 } }
I20240503 02:12:12.872898 29449 raft_consensus.cc:2932] T 4e5d560763774c0cba004753269a158c P 20b9275d1c1a4f6c81b8555d956cd7bc [term 3 LEADER]: Committing config change with OpId 3.9: config changed from index 7 to 9, VOTER 6c704dfc19814db996a1c0e363e760ef (127.22.108.6) evicted. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "20b9275d1c1a4f6c81b8555d956cd7bc" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 43455 } } }
I20240503 02:12:12.877164 24843 catalog_manager.cc:5484] T 4e5d560763774c0cba004753269a158c P 20b9275d1c1a4f6c81b8555d956cd7bc reported cstate change: config changed from index 7 to 9, VOTER 6c704dfc19814db996a1c0e363e760ef (127.22.108.6) evicted. New cstate: current_term: 3 leader_uuid: "20b9275d1c1a4f6c81b8555d956cd7bc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "20b9275d1c1a4f6c81b8555d956cd7bc" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 43455 } } }
I20240503 02:12:12.887213 26555 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e5d560763774c0cba004753269a158c with delete_type TABLET_DATA_TOMBSTONED (TS 6c704dfc19814db996a1c0e363e760ef not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:43946
I20240503 02:12:12.888152 29569 tablet_replica.cc:330] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef: stopping tablet replica
I20240503 02:12:12.888628 29569 raft_consensus.cc:2229] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:12.888909 29569 raft_consensus.cc:2258] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:12.891423 29569 ts_tablet_manager.cc:1875] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:12.898020 29569 ts_tablet_manager.cc:1888] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.8
I20240503 02:12:12.898167 29569 log.cc:1196] T 4e5d560763774c0cba004753269a158c P 6c704dfc19814db996a1c0e363e760ef: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal/wals/4e5d560763774c0cba004753269a158c
I20240503 02:12:12.898967 24835 catalog_manager.cc:4830] TS 6c704dfc19814db996a1c0e363e760ef (127.22.108.6:43731): tablet 4e5d560763774c0cba004753269a158c (table rebalance_test_table_3 [id=ac074aff80ae4832bb677cde4982aed7]) successfully deleted
I20240503 02:12:13.195127 26685 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:13.196390 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:13.209722 29058 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:13.210965 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:13.213788 24970 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:13.232900 26555 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:13.243731 25100 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:13.268577 26164 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:13.463325 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:13.465869 26685 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:13.485970 25100 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:13.488929 29058 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:13.502252 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:13.505249 24970 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:13.540553 26555 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:13.573921 26164 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:13.872841 26425 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:13.880654 26685 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:13.881268 26295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:13.898926 29058 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:13.921499 26555 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:13.927593 24970 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:13.939941 25100 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:13.964111 26164 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
5a79f2a86741450aa9992d60c94daf6f | 127.22.108.62:38409 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.108.20:37329 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
0163beb78bfd4d399a70ebacb8f9aa9c | 127.22.108.4:45401 | HEALTHY | <none> | 2 | 0
03afb03fe005462eb07ada37fe23c3cf | 127.22.108.7:34851 | HEALTHY | <none> | 2 | 0
1b2a71b48d4e40679504230af08279e9 | 127.22.108.5:40741 | HEALTHY | <none> | 2 | 0
20b9275d1c1a4f6c81b8555d956cd7bc | 127.22.108.8:43455 | HEALTHY | <none> | 2 | 0
22b0b1ae6c504301ad5170c6bdcd179e | 127.22.108.2:36213 | HEALTHY | <none> | 2 | 0
308436a6f7c7456b99971c666d3bd049 | 127.22.108.1:40941 | HEALTHY | <none> | 2 | 0
6c704dfc19814db996a1c0e363e760ef | 127.22.108.6:43731 | HEALTHY | <none> | 2 | 0
b95ea99cb72f49c6b42c7e8e293cfc27 | 127.22.108.3:36823 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.22.108.1 | experimental | 127.22.108.1:40941
local_ip_for_outbound_sockets | 127.22.108.2 | experimental | 127.22.108.2:36213
local_ip_for_outbound_sockets | 127.22.108.3 | experimental | 127.22.108.3:36823
local_ip_for_outbound_sockets | 127.22.108.4 | experimental | 127.22.108.4:45401
local_ip_for_outbound_sockets | 127.22.108.5 | experimental | 127.22.108.5:40741
local_ip_for_outbound_sockets | 127.22.108.6 | experimental | 127.22.108.6:43731
local_ip_for_outbound_sockets | 127.22.108.7 | experimental | 127.22.108.7:34851
local_ip_for_outbound_sockets | 127.22.108.8 | experimental | 127.22.108.8:43455
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.22.108.1:40941
server_dump_info_path | /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.22.108.2:36213
server_dump_info_path | /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.22.108.3:36823
server_dump_info_path | /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.22.108.4:45401
server_dump_info_path | /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.22.108.5:40741
server_dump_info_path | /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.22.108.6:43731
server_dump_info_path | /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.22.108.7:34851
server_dump_info_path | /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.22.108.8:43455
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.22.108.20:37329 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 02:12:14.036136 22960 log_verifier.cc:126] Checking tablet 4ce4d66aff0e48deb5513001f1d81963
I20240503 02:12:14.046530 22960 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 4ce4d66aff0e48deb5513001f1d81963
I20240503 02:12:14.046622 22960 log_verifier.cc:126] Checking tablet 4e5d560763774c0cba004753269a158c
I20240503 02:12:14.059403 22960 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 4e5d560763774c0cba004753269a158c
I20240503 02:12:14.059517 22960 log_verifier.cc:126] Checking tablet 742a25555073462faaabd4af58f42017
I20240503 02:12:14.069171 22960 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 742a25555073462faaabd4af58f42017
I20240503 02:12:14.069259 22960 log_verifier.cc:126] Checking tablet 7b671abb263d4848b7b909927b7df380
I20240503 02:12:14.079835 22960 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 7b671abb263d4848b7b909927b7df380
I20240503 02:12:14.079938 22960 log_verifier.cc:126] Checking tablet 7c0b16db9b77442b922e3b7bf1247878
I20240503 02:12:14.089964 22960 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 7c0b16db9b77442b922e3b7bf1247878
I20240503 02:12:14.090062 22960 log_verifier.cc:126] Checking tablet 88000b61106c4087948a9aa37ecc297f
I20240503 02:12:14.102469 22960 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 88000b61106c4087948a9aa37ecc297f
I20240503 02:12:14.102586 22960 log_verifier.cc:126] Checking tablet 923672905ffb452993c5dcf283f4a399
I20240503 02:12:14.112691 22960 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 923672905ffb452993c5dcf283f4a399
I20240503 02:12:14.112795 22960 log_verifier.cc:126] Checking tablet 92f95fea174e43c9bc3aa019d2191e51
I20240503 02:12:14.121865 22960 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 92f95fea174e43c9bc3aa019d2191e51
I20240503 02:12:14.121968 22960 log_verifier.cc:126] Checking tablet a141985c93da484494d5f82c478d0bc3
I20240503 02:12:14.131697 22960 log_verifier.cc:177] Verified matching terms for 6 ops in tablet a141985c93da484494d5f82c478d0bc3
I20240503 02:12:14.131804 22960 log_verifier.cc:126] Checking tablet af70ad6a7a8a4051beb408ead22d30e3
I20240503 02:12:14.140815 22960 log_verifier.cc:177] Verified matching terms for 3 ops in tablet af70ad6a7a8a4051beb408ead22d30e3
I20240503 02:12:14.140915 22960 log_verifier.cc:126] Checking tablet bb1a9efdfd96421ea17c41cfa8977f36
I20240503 02:12:14.151029 22960 log_verifier.cc:177] Verified matching terms for 6 ops in tablet bb1a9efdfd96421ea17c41cfa8977f36
I20240503 02:12:14.151130 22960 log_verifier.cc:126] Checking tablet e1093d22a1dc4e0f81fd3df39bf03265
I20240503 02:12:14.160841 22960 log_verifier.cc:177] Verified matching terms for 6 ops in tablet e1093d22a1dc4e0f81fd3df39bf03265
I20240503 02:12:14.160948 22960 log_verifier.cc:126] Checking tablet ed43f867d21d4bee9efc0b9bf4ee90c1
I20240503 02:12:14.170783 22960 log_verifier.cc:177] Verified matching terms for 6 ops in tablet ed43f867d21d4bee9efc0b9bf4ee90c1
I20240503 02:12:14.170900 22960 log_verifier.cc:126] Checking tablet ee1a68079bfa435ab820967fb894d3f1
I20240503 02:12:14.176265 22960 log_verifier.cc:177] Verified matching terms for 4 ops in tablet ee1a68079bfa435ab820967fb894d3f1
I20240503 02:12:14.176362 22960 log_verifier.cc:126] Checking tablet f6920bfd97624ad381fd72b688ab73f1
I20240503 02:12:14.186127 22960 log_verifier.cc:177] Verified matching terms for 6 ops in tablet f6920bfd97624ad381fd72b688ab73f1
I20240503 02:12:14.186232 22960 log_verifier.cc:126] Checking tablet fb78606eb2e74e19ab30bf7a119ea7e6
I20240503 02:12:14.196017 22960 log_verifier.cc:177] Verified matching terms for 3 ops in tablet fb78606eb2e74e19ab30bf7a119ea7e6
I20240503 02:12:14.196979 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 24908
I20240503 02:12:14.198426 25029 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:14.423032 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 24908
I20240503 02:12:14.460886 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 25039
I20240503 02:12:14.462059 25160 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:14.672112 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 25039
I20240503 02:12:14.707695 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 26102
I20240503 02:12:14.714093 26223 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:14.906644 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 26102
W20240503 02:12:14.939388 24836 connection.cc:535] server connection from 127.22.108.3:55009 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:12:14.939692 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 26233
I20240503 02:12:14.946213 26354 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:15.140048 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 26233
W20240503 02:12:15.171999 24833 connection.cc:535] server connection from 127.22.108.4:52989 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:12:15.172245 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 26363
I20240503 02:12:15.186544 26484 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:15.378391 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 26363
I20240503 02:12:15.410395 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 26493
I20240503 02:12:15.416677 26615 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:15.608137 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 26493
I20240503 02:12:15.640292 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 26623
I20240503 02:12:15.646656 26744 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:15.833109 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 26623
I20240503 02:12:15.863505 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 28996
I20240503 02:12:15.869619 29118 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:15.983456 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 28996
W20240503 02:12:16.003350 24836 connection.cc:535] server connection from 127.22.108.8:43859 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:12:16.003676 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 24818
I20240503 02:12:16.004711 24879 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:16.244251 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 24818
2024-05-03T02:12:16Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/0 (46720 ms)
[----------] 1 test from RebalanceParamTest (46720 ms total)
[----------] 1 test from DDLDuringRebalancingTest
[ RUN ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/0
I20240503 02:12:16.320607 22960 test_util.cc:268] Using random seed: 1067902025
I20240503 02:12:16.322424 22960 ts_itest-base.cc:112] Starting cluster with:
I20240503 02:12:16.322494 22960 ts_itest-base.cc:113] --------------
I20240503 02:12:16.322556 22960 ts_itest-base.cc:114] 8 tablet servers
I20240503 02:12:16.322604 22960 ts_itest-base.cc:115] 3 replicas per TS
I20240503 02:12:16.322666 22960 ts_itest-base.cc:116] --------------
2024-05-03T02:12:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:12:16Z Disabled control of system clock
I20240503 02:12:16.352531 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.108.62:40071
--webserver_interface=127.22.108.62
--webserver_port=0
--builtin_ntp_servers=127.22.108.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.108.62:40071
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240503 02:12:16.684237 29703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:16.684589 29703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:16.684726 29703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:16.692133 29703 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:12:16.692287 29703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:12:16.692389 29703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:16.692473 29703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:12:16.692540 29703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:12:16.701668 29703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.22.108.62:40071
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.108.62:40071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.108.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:16.703056 29703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:16.704942 29703 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:16.716615 29708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:16.717301 29709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:16.717955 29711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:16.718755 29703 server_base.cc:1034] running on GCE node
I20240503 02:12:16.719581 29703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:16.721115 29703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:16.722314 29703 hybrid_clock.cc:648] HybridClock initialized: now 1714702336722254 us; error 60 us; skew 500 ppm
I20240503 02:12:16.722658 29703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:16.724896 29703 webserver.cc:460] Webserver started at http://127.22.108.62:38635/ using document root <none> and password file <none>
I20240503 02:12:16.725682 29703 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:16.725797 29703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:16.726099 29703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:16.728868 29703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "e72f3141a78a4d0e811377c9c1f11f00"
format_stamp: "Formatted at 2024-05-03 02:12:16 on dist-test-slave-sc15"
I20240503 02:12:16.729763 29703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "e72f3141a78a4d0e811377c9c1f11f00"
format_stamp: "Formatted at 2024-05-03 02:12:16 on dist-test-slave-sc15"
I20240503 02:12:16.734977 29703 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20240503 02:12:16.738325 29717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:16.739727 29703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:12:16.739894 29703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e72f3141a78a4d0e811377c9c1f11f00"
format_stamp: "Formatted at 2024-05-03 02:12:16 on dist-test-slave-sc15"
I20240503 02:12:16.740103 29703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:16.784000 29703 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:16.784911 29703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:16.785212 29703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:16.807296 29703 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.62:40071
I20240503 02:12:16.807304 29768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.62:40071 every 8 connection(s)
I20240503 02:12:16.808704 29703 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 02:12:16.811266 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 29703
I20240503 02:12:16.811487 22960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 02:12:16.812997 29769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:16.825598 29769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Bootstrap starting.
I20240503 02:12:16.828536 29769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:16.829545 29769 log.cc:824] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:16.832012 29769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: No bootstrap required, opened a new log
I20240503 02:12:16.838119 29769 raft_consensus.cc:348] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e72f3141a78a4d0e811377c9c1f11f00" member_type: VOTER last_known_addr { host: "127.22.108.62" port: 40071 } }
I20240503 02:12:16.838508 29769 raft_consensus.cc:374] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:16.838629 29769 raft_consensus.cc:729] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72f3141a78a4d0e811377c9c1f11f00, State: Initialized, Role: FOLLOWER
I20240503 02:12:16.839287 29769 consensus_queue.cc:260] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e72f3141a78a4d0e811377c9c1f11f00" member_type: VOTER last_known_addr { host: "127.22.108.62" port: 40071 } }
I20240503 02:12:16.839502 29769 raft_consensus.cc:388] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:12:16.839625 29769 raft_consensus.cc:482] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:12:16.839769 29769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:16.841944 29769 raft_consensus.cc:504] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e72f3141a78a4d0e811377c9c1f11f00" member_type: VOTER last_known_addr { host: "127.22.108.62" port: 40071 } }
I20240503 02:12:16.842463 29769 leader_election.cc:302] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e72f3141a78a4d0e811377c9c1f11f00; no voters:
I20240503 02:12:16.843034 29769 leader_election.cc:288] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:12:16.843173 29774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:16.843809 29774 raft_consensus.cc:686] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00 [term 1 LEADER]: Becoming Leader. State: Replica: e72f3141a78a4d0e811377c9c1f11f00, State: Running, Role: LEADER
I20240503 02:12:16.844365 29774 consensus_queue.cc:237] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e72f3141a78a4d0e811377c9c1f11f00" member_type: VOTER last_known_addr { host: "127.22.108.62" port: 40071 } }
I20240503 02:12:16.845165 29769 sys_catalog.cc:564] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:12:16.847016 29776 sys_catalog.cc:455] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e72f3141a78a4d0e811377c9c1f11f00. Latest consensus state: current_term: 1 leader_uuid: "e72f3141a78a4d0e811377c9c1f11f00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e72f3141a78a4d0e811377c9c1f11f00" member_type: VOTER last_known_addr { host: "127.22.108.62" port: 40071 } } }
I20240503 02:12:16.847301 29776 sys_catalog.cc:458] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00 [sys.catalog]: This master's current role is: LEADER
I20240503 02:12:16.847180 29775 sys_catalog.cc:455] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e72f3141a78a4d0e811377c9c1f11f00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e72f3141a78a4d0e811377c9c1f11f00" member_type: VOTER last_known_addr { host: "127.22.108.62" port: 40071 } } }
I20240503 02:12:16.847522 29775 sys_catalog.cc:458] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00 [sys.catalog]: This master's current role is: LEADER
I20240503 02:12:16.848224 29779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:12:16.856949 29779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:12:16.865831 29779 catalog_manager.cc:1347] Generated new cluster ID: 5408211d8f594d34b975a0af20108168
I20240503 02:12:16.865960 29779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:12:16.925747 29779 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:12:16.927026 29779 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:12:16.943305 29779 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Generated new TSK 0
I20240503 02:12:16.944325 29779 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:12:16.965005 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.1:0
--local_ip_for_outbound_sockets=127.22.108.1
--webserver_interface=127.22.108.1
--webserver_port=0
--tserver_master_addrs=127.22.108.62:40071
--builtin_ntp_servers=127.22.108.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 02:12:17.303673 29793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:17.304010 29793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:17.304141 29793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:17.311913 29793 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:12:17.312103 29793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:17.312263 29793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.1
I20240503 02:12:17.321442 29793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.108.1
--webserver_port=0
--tserver_master_addrs=127.22.108.62:40071
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.1
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:17.322849 29793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:17.324719 29793 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:17.337004 29799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:17.337815 29801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:17.338500 29798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:17.339385 29793 server_base.cc:1034] running on GCE node
I20240503 02:12:17.339941 29793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:17.340842 29793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:17.342056 29793 hybrid_clock.cc:648] HybridClock initialized: now 1714702337341998 us; error 51 us; skew 500 ppm
I20240503 02:12:17.342370 29793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:17.344656 29793 webserver.cc:460] Webserver started at http://127.22.108.1:46363/ using document root <none> and password file <none>
I20240503 02:12:17.345531 29793 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:17.345649 29793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:17.345955 29793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:17.348784 29793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "16440e365430405fb10366ab70e2b429"
format_stamp: "Formatted at 2024-05-03 02:12:17 on dist-test-slave-sc15"
I20240503 02:12:17.349656 29793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "16440e365430405fb10366ab70e2b429"
format_stamp: "Formatted at 2024-05-03 02:12:17 on dist-test-slave-sc15"
I20240503 02:12:17.354786 29793 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240503 02:12:17.358243 29807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:17.359575 29793 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240503 02:12:17.359730 29793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "16440e365430405fb10366ab70e2b429"
format_stamp: "Formatted at 2024-05-03 02:12:17 on dist-test-slave-sc15"
I20240503 02:12:17.359920 29793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:17.382409 29793 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:17.383266 29793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:17.383544 29793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:17.384501 29793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:17.386031 29793 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:17.386128 29793 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:17.386236 29793 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:17.386317 29793 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:17.417665 29793 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.1:36719
I20240503 02:12:17.417714 29919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.1:36719 every 8 connection(s)
I20240503 02:12:17.418994 29793 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 02:12:17.426537 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 29793
I20240503 02:12:17.426738 22960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 02:12:17.430056 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.2:0
--local_ip_for_outbound_sockets=127.22.108.2
--webserver_interface=127.22.108.2
--webserver_port=0
--tserver_master_addrs=127.22.108.62:40071
--builtin_ntp_servers=127.22.108.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:12:17.431926 29920 heartbeater.cc:347] Connected to a master server at 127.22.108.62:40071
I20240503 02:12:17.432215 29920 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:17.432946 29920 heartbeater.cc:511] Master 127.22.108.62:40071 requested a full tablet report, sending...
I20240503 02:12:17.435232 29734 ts_manager.cc:194] Registered new tserver with Master: 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:17.437153 29734 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.1:54113
W20240503 02:12:17.763255 29924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:17.763610 29924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:17.763746 29924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:17.771391 29924 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:12:17.771522 29924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:17.771646 29924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.2
I20240503 02:12:17.780727 29924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.108.2
--webserver_port=0
--tserver_master_addrs=127.22.108.62:40071
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.2
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:17.782090 29924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:17.783816 29924 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:17.796017 29930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:17.796535 29932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:17.797139 29929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:17.798216 29924 server_base.cc:1034] running on GCE node
I20240503 02:12:17.798803 29924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:17.799713 29924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:17.800933 29924 hybrid_clock.cc:648] HybridClock initialized: now 1714702337800870 us; error 54 us; skew 500 ppm
I20240503 02:12:17.801256 29924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:17.803463 29924 webserver.cc:460] Webserver started at http://127.22.108.2:37049/ using document root <none> and password file <none>
I20240503 02:12:17.804196 29924 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:17.804293 29924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:17.804559 29924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:17.807464 29924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "155735c7f8b14997ac4c770aee2a9401"
format_stamp: "Formatted at 2024-05-03 02:12:17 on dist-test-slave-sc15"
I20240503 02:12:17.808477 29924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "155735c7f8b14997ac4c770aee2a9401"
format_stamp: "Formatted at 2024-05-03 02:12:17 on dist-test-slave-sc15"
I20240503 02:12:17.815043 29924 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.003s
I20240503 02:12:17.819324 29938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:17.821009 29924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:17.821208 29924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "155735c7f8b14997ac4c770aee2a9401"
format_stamp: "Formatted at 2024-05-03 02:12:17 on dist-test-slave-sc15"
I20240503 02:12:17.821425 29924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:17.839824 29924 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:17.840621 29924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:17.840870 29924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:17.841792 29924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:17.843257 29924 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:17.843343 29924 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:17.843423 29924 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:17.843479 29924 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:17.877259 29924 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.2:37873
I20240503 02:12:17.877329 30050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.2:37873 every 8 connection(s)
I20240503 02:12:17.878996 29924 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 02:12:17.881419 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 29924
I20240503 02:12:17.881614 22960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 02:12:17.884505 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.3:0
--local_ip_for_outbound_sockets=127.22.108.3
--webserver_interface=127.22.108.3
--webserver_port=0
--tserver_master_addrs=127.22.108.62:40071
--builtin_ntp_servers=127.22.108.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:12:17.891863 30051 heartbeater.cc:347] Connected to a master server at 127.22.108.62:40071
I20240503 02:12:17.892190 30051 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:17.893134 30051 heartbeater.cc:511] Master 127.22.108.62:40071 requested a full tablet report, sending...
I20240503 02:12:17.894620 29734 ts_manager.cc:194] Registered new tserver with Master: 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:12:17.895387 29734 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.2:35465
W20240503 02:12:18.217794 30054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:18.218082 30054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:18.218204 30054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:18.225858 30054 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:12:18.225983 30054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:18.226137 30054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.3
I20240503 02:12:18.235291 30054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.108.3
--webserver_port=0
--tserver_master_addrs=127.22.108.62:40071
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.3
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:18.236675 30054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:18.238543 30054 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:18.251008 30059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:18.251704 30062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:18.252458 30060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:18.253331 30054 server_base.cc:1034] running on GCE node
I20240503 02:12:18.253958 30054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:18.254949 30054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:18.256171 30054 hybrid_clock.cc:648] HybridClock initialized: now 1714702338256109 us; error 46 us; skew 500 ppm
I20240503 02:12:18.256516 30054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:18.258949 30054 webserver.cc:460] Webserver started at http://127.22.108.3:36233/ using document root <none> and password file <none>
I20240503 02:12:18.259730 30054 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:18.259843 30054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:18.260154 30054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:18.262867 30054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "a9094240860e43bfb129193f3bfd5be6"
format_stamp: "Formatted at 2024-05-03 02:12:18 on dist-test-slave-sc15"
I20240503 02:12:18.263700 30054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "a9094240860e43bfb129193f3bfd5be6"
format_stamp: "Formatted at 2024-05-03 02:12:18 on dist-test-slave-sc15"
I20240503 02:12:18.269109 30054 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20240503 02:12:18.272514 30068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:18.274031 30054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:12:18.274217 30054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "a9094240860e43bfb129193f3bfd5be6"
format_stamp: "Formatted at 2024-05-03 02:12:18 on dist-test-slave-sc15"
I20240503 02:12:18.274447 30054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:18.298532 30054 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:18.299466 30054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:18.299778 30054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:18.300805 30054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:18.302381 30054 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:18.302498 30054 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:18.302610 30054 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:18.302681 30054 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:18.332979 30054 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.3:42679
I20240503 02:12:18.333048 30180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.3:42679 every 8 connection(s)
I20240503 02:12:18.334331 30054 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:12:18.337879 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 30054
I20240503 02:12:18.338083 22960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 02:12:18.341514 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.4:0
--local_ip_for_outbound_sockets=127.22.108.4
--webserver_interface=127.22.108.4
--webserver_port=0
--tserver_master_addrs=127.22.108.62:40071
--builtin_ntp_servers=127.22.108.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:12:18.348137 30181 heartbeater.cc:347] Connected to a master server at 127.22.108.62:40071
I20240503 02:12:18.348467 30181 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:18.349258 30181 heartbeater.cc:511] Master 127.22.108.62:40071 requested a full tablet report, sending...
I20240503 02:12:18.350679 29734 ts_manager.cc:194] Registered new tserver with Master: a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:18.351419 29734 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.3:57409
I20240503 02:12:18.440062 29920 heartbeater.cc:503] Master 127.22.108.62:40071 was elected leader, sending a full tablet report...
W20240503 02:12:18.677495 30184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:18.677824 30184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:18.677959 30184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:18.685602 30184 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:12:18.685760 30184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:18.685921 30184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.4
I20240503 02:12:18.695048 30184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.108.4
--webserver_port=0
--tserver_master_addrs=127.22.108.62:40071
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.4
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:18.696447 30184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:18.698299 30184 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:18.711082 30189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:18.711916 30190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:18.712622 30192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:18.713608 30184 server_base.cc:1034] running on GCE node
I20240503 02:12:18.714294 30184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:18.715497 30184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:18.716727 30184 hybrid_clock.cc:648] HybridClock initialized: now 1714702338716668 us; error 56 us; skew 500 ppm
I20240503 02:12:18.717114 30184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:18.719553 30184 webserver.cc:460] Webserver started at http://127.22.108.4:40249/ using document root <none> and password file <none>
I20240503 02:12:18.720345 30184 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:18.720474 30184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:18.720835 30184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:18.723968 30184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "1f10f80f13c84e80b03eda7085a3b012"
format_stamp: "Formatted at 2024-05-03 02:12:18 on dist-test-slave-sc15"
I20240503 02:12:18.724846 30184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "1f10f80f13c84e80b03eda7085a3b012"
format_stamp: "Formatted at 2024-05-03 02:12:18 on dist-test-slave-sc15"
I20240503 02:12:18.730554 30184 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20240503 02:12:18.734409 30198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:18.735925 30184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:12:18.736109 30184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "1f10f80f13c84e80b03eda7085a3b012"
format_stamp: "Formatted at 2024-05-03 02:12:18 on dist-test-slave-sc15"
I20240503 02:12:18.736323 30184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:18.760531 30184 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:18.761485 30184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:18.761760 30184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:18.762713 30184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:18.764268 30184 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:18.764366 30184 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:18.764482 30184 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:18.764554 30184 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:18.796577 30184 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.4:33935
I20240503 02:12:18.796628 30310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.4:33935 every 8 connection(s)
I20240503 02:12:18.797983 30184 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:12:18.803200 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 30184
I20240503 02:12:18.803364 22960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 02:12:18.806430 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.5:0
--local_ip_for_outbound_sockets=127.22.108.5
--webserver_interface=127.22.108.5
--webserver_port=0
--tserver_master_addrs=127.22.108.62:40071
--builtin_ntp_servers=127.22.108.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:12:18.811517 30311 heartbeater.cc:347] Connected to a master server at 127.22.108.62:40071
I20240503 02:12:18.811878 30311 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:18.812796 30311 heartbeater.cc:511] Master 127.22.108.62:40071 requested a full tablet report, sending...
I20240503 02:12:18.814565 29734 ts_manager.cc:194] Registered new tserver with Master: 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:12:18.815869 29734 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.4:39103
I20240503 02:12:18.898401 30051 heartbeater.cc:503] Master 127.22.108.62:40071 was elected leader, sending a full tablet report...
W20240503 02:12:19.156153 30314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:19.156491 30314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:19.156643 30314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:19.164439 30314 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:12:19.164577 30314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:19.164752 30314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.5
I20240503 02:12:19.174206 30314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.108.5
--webserver_port=0
--tserver_master_addrs=127.22.108.62:40071
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.5
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:19.175591 30314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:19.177426 30314 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:19.190537 30320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:19.191062 30319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:19.191888 30322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:19.192778 30314 server_base.cc:1034] running on GCE node
I20240503 02:12:19.193466 30314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:19.194425 30314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:19.195629 30314 hybrid_clock.cc:648] HybridClock initialized: now 1714702339195580 us; error 42 us; skew 500 ppm
I20240503 02:12:19.195951 30314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:19.198316 30314 webserver.cc:460] Webserver started at http://127.22.108.5:40379/ using document root <none> and password file <none>
I20240503 02:12:19.199093 30314 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:19.199211 30314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:19.199524 30314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:19.204069 30314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "9a4d92bc25374fb3960fde7d10be309d"
format_stamp: "Formatted at 2024-05-03 02:12:19 on dist-test-slave-sc15"
I20240503 02:12:19.204902 30314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "9a4d92bc25374fb3960fde7d10be309d"
format_stamp: "Formatted at 2024-05-03 02:12:19 on dist-test-slave-sc15"
I20240503 02:12:19.210362 30314 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240503 02:12:19.214252 30328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:19.215817 30314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240503 02:12:19.216007 30314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "9a4d92bc25374fb3960fde7d10be309d"
format_stamp: "Formatted at 2024-05-03 02:12:19 on dist-test-slave-sc15"
I20240503 02:12:19.216226 30314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:19.239096 30314 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:19.240006 30314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:19.240272 30314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:19.241298 30314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:19.242813 30314 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:19.242911 30314 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:19.243018 30314 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:19.243091 30314 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:19.275023 30314 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.5:35595
I20240503 02:12:19.275077 30440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.5:35595 every 8 connection(s)
I20240503 02:12:19.276374 30314 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:12:19.279981 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 30314
I20240503 02:12:19.280174 22960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 02:12:19.283121 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.6:0
--local_ip_for_outbound_sockets=127.22.108.6
--webserver_interface=127.22.108.6
--webserver_port=0
--tserver_master_addrs=127.22.108.62:40071
--builtin_ntp_servers=127.22.108.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:12:19.289577 30441 heartbeater.cc:347] Connected to a master server at 127.22.108.62:40071
I20240503 02:12:19.289937 30441 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:19.290894 30441 heartbeater.cc:511] Master 127.22.108.62:40071 requested a full tablet report, sending...
I20240503 02:12:19.292533 29734 ts_manager.cc:194] Registered new tserver with Master: 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:12:19.293329 29734 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.5:58063
I20240503 02:12:19.353797 30181 heartbeater.cc:503] Master 127.22.108.62:40071 was elected leader, sending a full tablet report...
W20240503 02:12:19.621744 30444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:19.622061 30444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:19.622193 30444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:19.629858 30444 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:12:19.630003 30444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:19.630162 30444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.6
I20240503 02:12:19.639336 30444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.22.108.6
--webserver_port=0
--tserver_master_addrs=127.22.108.62:40071
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.6
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:19.640738 30444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:19.642634 30444 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:19.655203 30449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:19.656222 30444 server_base.cc:1034] running on GCE node
W20240503 02:12:19.656731 30452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:19.657366 30450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:19.658008 30444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:19.658922 30444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:19.660157 30444 hybrid_clock.cc:648] HybridClock initialized: now 1714702339660097 us; error 52 us; skew 500 ppm
I20240503 02:12:19.660486 30444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:19.662801 30444 webserver.cc:460] Webserver started at http://127.22.108.6:35081/ using document root <none> and password file <none>
I20240503 02:12:19.663540 30444 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:19.663654 30444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:19.663965 30444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:19.666671 30444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "8a666e798fe8477d8dd1e51b385d5ead"
format_stamp: "Formatted at 2024-05-03 02:12:19 on dist-test-slave-sc15"
I20240503 02:12:19.667469 30444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "8a666e798fe8477d8dd1e51b385d5ead"
format_stamp: "Formatted at 2024-05-03 02:12:19 on dist-test-slave-sc15"
I20240503 02:12:19.672670 30444 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 02:12:19.676061 30458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:19.677384 30444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240503 02:12:19.677527 30444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "8a666e798fe8477d8dd1e51b385d5ead"
format_stamp: "Formatted at 2024-05-03 02:12:19 on dist-test-slave-sc15"
I20240503 02:12:19.677711 30444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:19.707806 30444 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:19.708710 30444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:19.708979 30444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:19.710029 30444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:19.711504 30444 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:19.711602 30444 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:19.711732 30444 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:19.711803 30444 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:19.742542 30444 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.6:39975
I20240503 02:12:19.742607 30570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.6:39975 every 8 connection(s)
I20240503 02:12:19.743929 30444 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 02:12:19.746217 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 30444
I20240503 02:12:19.746398 22960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 02:12:19.749997 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.7:0
--local_ip_for_outbound_sockets=127.22.108.7
--webserver_interface=127.22.108.7
--webserver_port=0
--tserver_master_addrs=127.22.108.62:40071
--builtin_ntp_servers=127.22.108.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:12:19.758062 30571 heartbeater.cc:347] Connected to a master server at 127.22.108.62:40071
I20240503 02:12:19.758416 30571 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:19.759372 30571 heartbeater.cc:511] Master 127.22.108.62:40071 requested a full tablet report, sending...
I20240503 02:12:19.760982 29734 ts_manager.cc:194] Registered new tserver with Master: 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:12:19.762192 29734 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.6:39129
I20240503 02:12:19.818531 30311 heartbeater.cc:503] Master 127.22.108.62:40071 was elected leader, sending a full tablet report...
W20240503 02:12:20.094462 30574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:20.094769 30574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:20.094902 30574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:20.102402 30574 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:12:20.102545 30574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:20.102710 30574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.7
I20240503 02:12:20.111970 30574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.22.108.7
--webserver_port=0
--tserver_master_addrs=127.22.108.62:40071
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.7
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:20.113422 30574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:20.115217 30574 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 02:12:20.128702 30574 server_base.cc:1034] running on GCE node
W20240503 02:12:20.129315 30580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:20.130107 30582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:20.130159 30579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:20.130934 30574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:20.131889 30574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:20.133091 30574 hybrid_clock.cc:648] HybridClock initialized: now 1714702340133038 us; error 49 us; skew 500 ppm
I20240503 02:12:20.133409 30574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:20.135728 30574 webserver.cc:460] Webserver started at http://127.22.108.7:40935/ using document root <none> and password file <none>
I20240503 02:12:20.136497 30574 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:20.136615 30574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:20.136926 30574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:20.139596 30574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "fbbc2be6382e47908ecb7c68c367bd26"
format_stamp: "Formatted at 2024-05-03 02:12:20 on dist-test-slave-sc15"
I20240503 02:12:20.140389 30574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "fbbc2be6382e47908ecb7c68c367bd26"
format_stamp: "Formatted at 2024-05-03 02:12:20 on dist-test-slave-sc15"
I20240503 02:12:20.145609 30574 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240503 02:12:20.148909 30588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:20.150300 30574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:12:20.150467 30574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "fbbc2be6382e47908ecb7c68c367bd26"
format_stamp: "Formatted at 2024-05-03 02:12:20 on dist-test-slave-sc15"
I20240503 02:12:20.150658 30574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:20.173962 30574 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:20.174872 30574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:20.175155 30574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:20.176175 30574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:20.177775 30574 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:20.177878 30574 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:20.177986 30574 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:20.178056 30574 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:20.209347 30574 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.7:44241
I20240503 02:12:20.209407 30700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.7:44241 every 8 connection(s)
I20240503 02:12:20.210718 30574 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 02:12:20.215309 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 30574
I20240503 02:12:20.215540 22960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 02:12:20.219202 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.8:0
--local_ip_for_outbound_sockets=127.22.108.8
--webserver_interface=127.22.108.8
--webserver_port=0
--tserver_master_addrs=127.22.108.62:40071
--builtin_ntp_servers=127.22.108.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:12:20.230695 30701 heartbeater.cc:347] Connected to a master server at 127.22.108.62:40071
I20240503 02:12:20.231031 30701 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:20.231863 30701 heartbeater.cc:511] Master 127.22.108.62:40071 requested a full tablet report, sending...
I20240503 02:12:20.233381 29734 ts_manager.cc:194] Registered new tserver with Master: fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:12:20.234194 29734 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.7:47327
I20240503 02:12:20.296489 30441 heartbeater.cc:503] Master 127.22.108.62:40071 was elected leader, sending a full tablet report...
W20240503 02:12:20.564837 30704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:20.565155 30704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:20.565271 30704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:20.572805 30704 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:12:20.572925 30704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:20.573067 30704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.8
I20240503 02:12:20.582113 30704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.22.108.8
--webserver_port=0
--tserver_master_addrs=127.22.108.62:40071
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.8
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:20.583422 30704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:20.585134 30704 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:20.597409 30709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:20.597929 30710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:20.598884 30712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:20.599774 30704 server_base.cc:1034] running on GCE node
I20240503 02:12:20.600366 30704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:20.601356 30704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:20.602581 30704 hybrid_clock.cc:648] HybridClock initialized: now 1714702340602526 us; error 49 us; skew 500 ppm
I20240503 02:12:20.602888 30704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:20.605118 30704 webserver.cc:460] Webserver started at http://127.22.108.8:46641/ using document root <none> and password file <none>
I20240503 02:12:20.605875 30704 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:20.605989 30704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:20.606288 30704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:20.608896 30704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
format_stamp: "Formatted at 2024-05-03 02:12:20 on dist-test-slave-sc15"
I20240503 02:12:20.609747 30704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
format_stamp: "Formatted at 2024-05-03 02:12:20 on dist-test-slave-sc15"
I20240503 02:12:20.615031 30704 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20240503 02:12:20.618544 30718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:20.619900 30704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 02:12:20.620054 30704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
format_stamp: "Formatted at 2024-05-03 02:12:20 on dist-test-slave-sc15"
I20240503 02:12:20.620255 30704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:20.650929 30704 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:20.651847 30704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:20.652122 30704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:20.653160 30704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:20.654889 30704 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:20.654990 30704 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:20.655108 30704 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:20.655185 30704 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:20.686960 30704 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.8:42233
I20240503 02:12:20.687018 30830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.8:42233 every 8 connection(s)
I20240503 02:12:20.688315 30704 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 02:12:20.693260 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 30704
I20240503 02:12:20.693466 22960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 02:12:20.701423 30831 heartbeater.cc:347] Connected to a master server at 127.22.108.62:40071
I20240503 02:12:20.701697 30831 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:20.702379 30831 heartbeater.cc:511] Master 127.22.108.62:40071 requested a full tablet report, sending...
I20240503 02:12:20.703610 29734 ts_manager.cc:194] Registered new tserver with Master: f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:12:20.704365 29734 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.8:37335
I20240503 02:12:20.709817 22960 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 02:12:20.726760 22960 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 02:12:20.726909 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 30314
I20240503 02:12:20.735314 30436 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:20.765501 30571 heartbeater.cc:503] Master 127.22.108.62:40071 was elected leader, sending a full tablet report...
I20240503 02:12:20.833918 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 30314
I20240503 02:12:20.850222 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 30444
I20240503 02:12:20.858369 30566 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:20.941560 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 30444
I20240503 02:12:20.958117 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 30574
I20240503 02:12:20.966172 30696 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:21.045928 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 30574
I20240503 02:12:21.062019 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 30704
I20240503 02:12:21.070986 30826 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:21.152379 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 30704
I20240503 02:12:24.920681 29734 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:24.957981 30245 tablet_service.cc:1467] Processing CreateTablet for tablet c5bb83b2a415451aa6909503a47662dd (DEFAULT_TABLE table=rebalance_test_table_0 [id=c53bfcf133b44207890453c637ba13d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.961683 30115 tablet_service.cc:1467] Processing CreateTablet for tablet 006b2e20b1b04b18bee60fcd822949c7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c53bfcf133b44207890453c637ba13d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.961896 30116 tablet_service.cc:1467] Processing CreateTablet for tablet c7f85d7037ce49a7b0bd9003d8977a5a (DEFAULT_TABLE table=rebalance_test_table_0 [id=c53bfcf133b44207890453c637ba13d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.963340 30115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 006b2e20b1b04b18bee60fcd822949c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.962893 29854 tablet_service.cc:1467] Processing CreateTablet for tablet c5bb83b2a415451aa6909503a47662dd (DEFAULT_TABLE table=rebalance_test_table_0 [id=c53bfcf133b44207890453c637ba13d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.964404 29854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5bb83b2a415451aa6909503a47662dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.964787 29855 tablet_service.cc:1467] Processing CreateTablet for tablet 006b2e20b1b04b18bee60fcd822949c7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c53bfcf133b44207890453c637ba13d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.964480 29986 tablet_service.cc:1467] Processing CreateTablet for tablet c7f85d7037ce49a7b0bd9003d8977a5a (DEFAULT_TABLE table=rebalance_test_table_0 [id=c53bfcf133b44207890453c637ba13d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.965319 29855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 006b2e20b1b04b18bee60fcd822949c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.965669 30116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7f85d7037ce49a7b0bd9003d8977a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.965878 29986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7f85d7037ce49a7b0bd9003d8977a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.965914 29984 tablet_service.cc:1467] Processing CreateTablet for tablet c5bb83b2a415451aa6909503a47662dd (DEFAULT_TABLE table=rebalance_test_table_0 [id=c53bfcf133b44207890453c637ba13d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.966337 29984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5bb83b2a415451aa6909503a47662dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.966899 29985 tablet_service.cc:1467] Processing CreateTablet for tablet 006b2e20b1b04b18bee60fcd822949c7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c53bfcf133b44207890453c637ba13d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.967360 29985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 006b2e20b1b04b18bee60fcd822949c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.969141 30245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5bb83b2a415451aa6909503a47662dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.968115 30246 tablet_service.cc:1467] Processing CreateTablet for tablet c7f85d7037ce49a7b0bd9003d8977a5a (DEFAULT_TABLE table=rebalance_test_table_0 [id=c53bfcf133b44207890453c637ba13d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:24.973392 30246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7f85d7037ce49a7b0bd9003d8977a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.987412 30861 tablet_bootstrap.cc:492] T 006b2e20b1b04b18bee60fcd822949c7 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:24.993093 30861 tablet_bootstrap.cc:654] T 006b2e20b1b04b18bee60fcd822949c7 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:24.994086 30862 tablet_bootstrap.cc:492] T c7f85d7037ce49a7b0bd9003d8977a5a P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:24.996815 30862 tablet_bootstrap.cc:654] T c7f85d7037ce49a7b0bd9003d8977a5a P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:24.998242 30861 log.cc:824] T 006b2e20b1b04b18bee60fcd822949c7 P 16440e365430405fb10366ab70e2b429: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:24.998241 30862 log.cc:824] T c7f85d7037ce49a7b0bd9003d8977a5a P a9094240860e43bfb129193f3bfd5be6: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:25.001538 30860 tablet_bootstrap.cc:492] T c7f85d7037ce49a7b0bd9003d8977a5a P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:25.005362 30860 tablet_bootstrap.cc:654] T c7f85d7037ce49a7b0bd9003d8977a5a P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:25.007027 30865 tablet_bootstrap.cc:492] T c5bb83b2a415451aa6909503a47662dd P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:25.008700 30860 log.cc:824] T c7f85d7037ce49a7b0bd9003d8977a5a P 155735c7f8b14997ac4c770aee2a9401: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:25.009127 30861 tablet_bootstrap.cc:492] T 006b2e20b1b04b18bee60fcd822949c7 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:25.009431 30861 ts_tablet_manager.cc:1367] T 006b2e20b1b04b18bee60fcd822949c7 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.004s
I20240503 02:12:25.010619 30865 tablet_bootstrap.cc:654] T c5bb83b2a415451aa6909503a47662dd P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:25.011421 30862 tablet_bootstrap.cc:492] T c7f85d7037ce49a7b0bd9003d8977a5a P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:25.011780 30862 ts_tablet_manager.cc:1367] T c7f85d7037ce49a7b0bd9003d8977a5a P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.000s
I20240503 02:12:25.011857 30865 log.cc:824] T c5bb83b2a415451aa6909503a47662dd P 1f10f80f13c84e80b03eda7085a3b012: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:25.021705 30861 raft_consensus.cc:348] T 006b2e20b1b04b18bee60fcd822949c7 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:25.022434 30861 raft_consensus.cc:374] T 006b2e20b1b04b18bee60fcd822949c7 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:25.022675 30861 raft_consensus.cc:729] T 006b2e20b1b04b18bee60fcd822949c7 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:25.023913 30861 consensus_queue.cc:260] T 006b2e20b1b04b18bee60fcd822949c7 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:25.024809 30862 raft_consensus.cc:348] T c7f85d7037ce49a7b0bd9003d8977a5a P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:25.025406 30862 raft_consensus.cc:374] T c7f85d7037ce49a7b0bd9003d8977a5a P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:25.025560 30862 raft_consensus.cc:729] T c7f85d7037ce49a7b0bd9003d8977a5a P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:25.026428 30862 consensus_queue.cc:260] T c7f85d7037ce49a7b0bd9003d8977a5a P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:25.027714 30860 tablet_bootstrap.cc:492] T c7f85d7037ce49a7b0bd9003d8977a5a P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:25.027714 30865 tablet_bootstrap.cc:492] T c5bb83b2a415451aa6909503a47662dd P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:25.028019 30865 ts_tablet_manager.cc:1367] T c5bb83b2a415451aa6909503a47662dd P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.009s
I20240503 02:12:25.029570 30862 ts_tablet_manager.cc:1398] T c7f85d7037ce49a7b0bd9003d8977a5a P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.018s user 0.019s sys 0.000s
I20240503 02:12:25.030040 30862 tablet_bootstrap.cc:492] T 006b2e20b1b04b18bee60fcd822949c7 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:25.031586 30861 ts_tablet_manager.cc:1398] T 006b2e20b1b04b18bee60fcd822949c7 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.022s user 0.011s sys 0.004s
I20240503 02:12:25.031785 30862 tablet_bootstrap.cc:654] T 006b2e20b1b04b18bee60fcd822949c7 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:25.032068 30861 tablet_bootstrap.cc:492] T c5bb83b2a415451aa6909503a47662dd P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:25.033257 30868 raft_consensus.cc:482] T 006b2e20b1b04b18bee60fcd822949c7 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:25.033489 30868 raft_consensus.cc:504] T 006b2e20b1b04b18bee60fcd822949c7 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:25.035549 30868 leader_election.cc:288] T 006b2e20b1b04b18bee60fcd822949c7 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:25.036336 30861 tablet_bootstrap.cc:654] T c5bb83b2a415451aa6909503a47662dd P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:25.037303 30865 raft_consensus.cc:348] T c5bb83b2a415451aa6909503a47662dd P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:25.037892 30865 raft_consensus.cc:374] T c5bb83b2a415451aa6909503a47662dd P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:25.038045 30865 raft_consensus.cc:729] T c5bb83b2a415451aa6909503a47662dd P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:25.039009 30865 consensus_queue.cc:260] T c5bb83b2a415451aa6909503a47662dd P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:25.040243 30862 tablet_bootstrap.cc:492] T 006b2e20b1b04b18bee60fcd822949c7 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:25.040521 30862 ts_tablet_manager.cc:1367] T 006b2e20b1b04b18bee60fcd822949c7 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20240503 02:12:25.041797 30862 raft_consensus.cc:348] T 006b2e20b1b04b18bee60fcd822949c7 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:25.042171 30862 raft_consensus.cc:374] T 006b2e20b1b04b18bee60fcd822949c7 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:25.042330 30862 raft_consensus.cc:729] T 006b2e20b1b04b18bee60fcd822949c7 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:25.042778 30862 consensus_queue.cc:260] T 006b2e20b1b04b18bee60fcd822949c7 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:25.043591 30862 ts_tablet_manager.cc:1398] T 006b2e20b1b04b18bee60fcd822949c7 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:25.045598 30860 ts_tablet_manager.cc:1367] T c7f85d7037ce49a7b0bd9003d8977a5a P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.019s
I20240503 02:12:25.047757 30861 tablet_bootstrap.cc:492] T c5bb83b2a415451aa6909503a47662dd P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:25.049389 30861 ts_tablet_manager.cc:1367] T c5bb83b2a415451aa6909503a47662dd P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.002s
I20240503 02:12:25.050621 30861 raft_consensus.cc:348] T c5bb83b2a415451aa6909503a47662dd P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:25.051026 30861 raft_consensus.cc:374] T c5bb83b2a415451aa6909503a47662dd P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:25.051179 30861 raft_consensus.cc:729] T c5bb83b2a415451aa6909503a47662dd P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:25.051853 30865 ts_tablet_manager.cc:1398] T c5bb83b2a415451aa6909503a47662dd P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.024s user 0.013s sys 0.006s
I20240503 02:12:25.051947 30861 consensus_queue.cc:260] T c5bb83b2a415451aa6909503a47662dd P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:25.052546 30861 ts_tablet_manager.cc:1398] T c5bb83b2a415451aa6909503a47662dd P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:12:25.053229 30865 tablet_bootstrap.cc:492] T c7f85d7037ce49a7b0bd9003d8977a5a P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
W20240503 02:12:25.054054 30312 tablet.cc:2369] T c5bb83b2a415451aa6909503a47662dd P 1f10f80f13c84e80b03eda7085a3b012: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:12:25.055091 30865 tablet_bootstrap.cc:654] T c7f85d7037ce49a7b0bd9003d8977a5a P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:25.056574 30860 raft_consensus.cc:348] T c7f85d7037ce49a7b0bd9003d8977a5a P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:25.057263 30860 raft_consensus.cc:374] T c7f85d7037ce49a7b0bd9003d8977a5a P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:25.057417 30860 raft_consensus.cc:729] T c7f85d7037ce49a7b0bd9003d8977a5a P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:25.058207 30860 consensus_queue.cc:260] T c7f85d7037ce49a7b0bd9003d8977a5a P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:25.060809 30136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006b2e20b1b04b18bee60fcd822949c7" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:12:25.061393 30136 raft_consensus.cc:2454] T 006b2e20b1b04b18bee60fcd822949c7 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:12:25.062178 30860 ts_tablet_manager.cc:1398] T c7f85d7037ce49a7b0bd9003d8977a5a P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.016s user 0.009s sys 0.004s
I20240503 02:12:25.062167 29808 leader_election.cc:302] T 006b2e20b1b04b18bee60fcd822949c7 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:25.062693 30860 tablet_bootstrap.cc:492] T c5bb83b2a415451aa6909503a47662dd P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:25.063246 30865 tablet_bootstrap.cc:492] T c7f85d7037ce49a7b0bd9003d8977a5a P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:25.063210 30868 raft_consensus.cc:2781] T 006b2e20b1b04b18bee60fcd822949c7 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:25.063354 30868 raft_consensus.cc:482] T 006b2e20b1b04b18bee60fcd822949c7 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:25.063453 30865 ts_tablet_manager.cc:1367] T c7f85d7037ce49a7b0bd9003d8977a5a P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20240503 02:12:25.063485 30868 raft_consensus.cc:3037] T 006b2e20b1b04b18bee60fcd822949c7 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:25.064500 30860 tablet_bootstrap.cc:654] T c5bb83b2a415451aa6909503a47662dd P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:25.064612 30865 raft_consensus.cc:348] T c7f85d7037ce49a7b0bd9003d8977a5a P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:25.064944 30865 raft_consensus.cc:374] T c7f85d7037ce49a7b0bd9003d8977a5a P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:25.065114 30865 raft_consensus.cc:729] T c7f85d7037ce49a7b0bd9003d8977a5a P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:25.065420 30865 consensus_queue.cc:260] T c7f85d7037ce49a7b0bd9003d8977a5a P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:25.066203 30865 ts_tablet_manager.cc:1398] T c7f85d7037ce49a7b0bd9003d8977a5a P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:25.066895 30868 raft_consensus.cc:504] T 006b2e20b1b04b18bee60fcd822949c7 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:25.067791 30868 leader_election.cc:288] T 006b2e20b1b04b18bee60fcd822949c7 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Requested vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:25.068717 30136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006b2e20b1b04b18bee60fcd822949c7" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:12:25.068955 30136 raft_consensus.cc:3037] T 006b2e20b1b04b18bee60fcd822949c7 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:25.070153 30860 tablet_bootstrap.cc:492] T c5bb83b2a415451aa6909503a47662dd P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:25.071385 30006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006b2e20b1b04b18bee60fcd822949c7" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:12:25.071780 30860 ts_tablet_manager.cc:1367] T c5bb83b2a415451aa6909503a47662dd P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20240503 02:12:25.072033 30136 raft_consensus.cc:2454] T 006b2e20b1b04b18bee60fcd822949c7 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:12:25.072468 29808 leader_election.cc:302] T 006b2e20b1b04b18bee60fcd822949c7 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:25.072576 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006b2e20b1b04b18bee60fcd822949c7" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:12:25.072935 30868 raft_consensus.cc:2781] T 006b2e20b1b04b18bee60fcd822949c7 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:25.072963 30860 raft_consensus.cc:348] T c5bb83b2a415451aa6909503a47662dd P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
W20240503 02:12:25.073122 29809 leader_election.cc:341] T 006b2e20b1b04b18bee60fcd822949c7 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:25.073248 30860 raft_consensus.cc:374] T c5bb83b2a415451aa6909503a47662dd P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:25.073335 30860 raft_consensus.cc:729] T c5bb83b2a415451aa6909503a47662dd P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:25.073493 30868 raft_consensus.cc:686] T 006b2e20b1b04b18bee60fcd822949c7 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Becoming Leader. State: Replica: 16440e365430405fb10366ab70e2b429, State: Running, Role: LEADER
I20240503 02:12:25.073603 30860 consensus_queue.cc:260] T c5bb83b2a415451aa6909503a47662dd P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
W20240503 02:12:25.073750 29809 leader_election.cc:341] T 006b2e20b1b04b18bee60fcd822949c7 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:25.074007 30868 consensus_queue.cc:237] T 006b2e20b1b04b18bee60fcd822949c7 P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:25.074203 30860 ts_tablet_manager.cc:1398] T c5bb83b2a415451aa6909503a47662dd P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:25.074476 30860 tablet_bootstrap.cc:492] T 006b2e20b1b04b18bee60fcd822949c7 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:25.075875 30860 tablet_bootstrap.cc:654] T 006b2e20b1b04b18bee60fcd822949c7 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:25.078142 30860 tablet_bootstrap.cc:492] T 006b2e20b1b04b18bee60fcd822949c7 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:25.078298 30860 ts_tablet_manager.cc:1367] T 006b2e20b1b04b18bee60fcd822949c7 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:12:25.078855 30860 raft_consensus.cc:348] T 006b2e20b1b04b18bee60fcd822949c7 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:25.078356 29733 catalog_manager.cc:5484] T 006b2e20b1b04b18bee60fcd822949c7 P 16440e365430405fb10366ab70e2b429 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16440e365430405fb10366ab70e2b429 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:12:25.079030 30860 raft_consensus.cc:374] T 006b2e20b1b04b18bee60fcd822949c7 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:25.079104 30860 raft_consensus.cc:729] T 006b2e20b1b04b18bee60fcd822949c7 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:25.079344 30860 consensus_queue.cc:260] T 006b2e20b1b04b18bee60fcd822949c7 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:25.079826 30860 ts_tablet_manager.cc:1398] T 006b2e20b1b04b18bee60fcd822949c7 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240503 02:12:25.091447 30182 tablet.cc:2369] T 006b2e20b1b04b18bee60fcd822949c7 P a9094240860e43bfb129193f3bfd5be6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:12:25.136282 30052 tablet.cc:2369] T 006b2e20b1b04b18bee60fcd822949c7 P 155735c7f8b14997ac4c770aee2a9401: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:12:25.165244 30876 raft_consensus.cc:482] T 006b2e20b1b04b18bee60fcd822949c7 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:25.165539 30876 raft_consensus.cc:504] T 006b2e20b1b04b18bee60fcd822949c7 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:25.167268 30876 leader_election.cc:288] T 006b2e20b1b04b18bee60fcd822949c7 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:25.174166 30136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006b2e20b1b04b18bee60fcd822949c7" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:12:25.174341 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006b2e20b1b04b18bee60fcd822949c7" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:25.174515 30136 raft_consensus.cc:2379] T 006b2e20b1b04b18bee60fcd822949c7 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 155735c7f8b14997ac4c770aee2a9401 in current term 1: Already voted for candidate 16440e365430405fb10366ab70e2b429 in this term.
I20240503 02:12:25.175344 29939 leader_election.cc:302] T 006b2e20b1b04b18bee60fcd822949c7 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401; no voters: 16440e365430405fb10366ab70e2b429, a9094240860e43bfb129193f3bfd5be6
I20240503 02:12:25.175753 30876 raft_consensus.cc:3037] T 006b2e20b1b04b18bee60fcd822949c7 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:25.176213 29921 tablet.cc:2369] T 006b2e20b1b04b18bee60fcd822949c7 P 16440e365430405fb10366ab70e2b429: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:12:25.177872 30876 raft_consensus.cc:2726] T 006b2e20b1b04b18bee60fcd822949c7 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:25.370803 30874 raft_consensus.cc:482] T c5bb83b2a415451aa6909503a47662dd P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:25.371064 30874 raft_consensus.cc:504] T c5bb83b2a415451aa6909503a47662dd P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:25.372859 30874 leader_election.cc:288] T c5bb83b2a415451aa6909503a47662dd P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:25.379797 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5bb83b2a415451aa6909503a47662dd" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:12:25.379797 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5bb83b2a415451aa6909503a47662dd" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:25.380084 30005 raft_consensus.cc:2454] T c5bb83b2a415451aa6909503a47662dd P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:25.380095 29875 raft_consensus.cc:2454] T c5bb83b2a415451aa6909503a47662dd P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:25.380635 30202 leader_election.cc:302] T c5bb83b2a415451aa6909503a47662dd P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:25.381052 30874 raft_consensus.cc:2781] T c5bb83b2a415451aa6909503a47662dd P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:25.381201 30874 raft_consensus.cc:482] T c5bb83b2a415451aa6909503a47662dd P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:25.381304 30874 raft_consensus.cc:3037] T c5bb83b2a415451aa6909503a47662dd P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:25.383352 30874 raft_consensus.cc:504] T c5bb83b2a415451aa6909503a47662dd P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:25.384013 30874 leader_election.cc:288] T c5bb83b2a415451aa6909503a47662dd P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Requested vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:25.384397 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5bb83b2a415451aa6909503a47662dd" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:12:25.384495 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5bb83b2a415451aa6909503a47662dd" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:12:25.384601 30005 raft_consensus.cc:3037] T c5bb83b2a415451aa6909503a47662dd P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:25.384725 29875 raft_consensus.cc:3037] T c5bb83b2a415451aa6909503a47662dd P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:25.386675 30005 raft_consensus.cc:2454] T c5bb83b2a415451aa6909503a47662dd P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:25.386687 29875 raft_consensus.cc:2454] T c5bb83b2a415451aa6909503a47662dd P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:25.387080 30202 leader_election.cc:302] T c5bb83b2a415451aa6909503a47662dd P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:25.387462 30874 raft_consensus.cc:2781] T c5bb83b2a415451aa6909503a47662dd P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:25.387849 30874 raft_consensus.cc:686] T c5bb83b2a415451aa6909503a47662dd P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Becoming Leader. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Running, Role: LEADER
I20240503 02:12:25.388283 30874 consensus_queue.cc:237] T c5bb83b2a415451aa6909503a47662dd P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:25.392681 29734 catalog_manager.cc:5484] T c5bb83b2a415451aa6909503a47662dd P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4). New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:12:25.493700 30869 raft_consensus.cc:482] T c7f85d7037ce49a7b0bd9003d8977a5a P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:25.493916 30869 raft_consensus.cc:504] T c7f85d7037ce49a7b0bd9003d8977a5a P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:25.495574 30869 leader_election.cc:288] T c7f85d7037ce49a7b0bd9003d8977a5a P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:12:25.497491 30874 raft_consensus.cc:482] T c7f85d7037ce49a7b0bd9003d8977a5a P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:25.497711 30874 raft_consensus.cc:504] T c7f85d7037ce49a7b0bd9003d8977a5a P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:25.498584 30874 leader_election.cc:288] T c7f85d7037ce49a7b0bd9003d8977a5a P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:12:25.499555 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7f85d7037ce49a7b0bd9003d8977a5a" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:12:25.499814 30005 raft_consensus.cc:2454] T c7f85d7037ce49a7b0bd9003d8977a5a P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:25.500299 30200 leader_election.cc:302] T c7f85d7037ce49a7b0bd9003d8977a5a P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:25.500681 30874 raft_consensus.cc:2781] T c7f85d7037ce49a7b0bd9003d8977a5a P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:25.500790 30874 raft_consensus.cc:482] T c7f85d7037ce49a7b0bd9003d8977a5a P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:25.500900 30874 raft_consensus.cc:3037] T c7f85d7037ce49a7b0bd9003d8977a5a P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:25.504098 30874 raft_consensus.cc:504] T c7f85d7037ce49a7b0bd9003d8977a5a P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:25.504830 30874 leader_election.cc:288] T c7f85d7037ce49a7b0bd9003d8977a5a P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Requested vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:12:25.505342 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7f85d7037ce49a7b0bd9003d8977a5a" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:12:25.505452 30006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7f85d7037ce49a7b0bd9003d8977a5a" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:12:25.505597 30005 raft_consensus.cc:2454] T c7f85d7037ce49a7b0bd9003d8977a5a P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 0.
I20240503 02:12:25.506121 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7f85d7037ce49a7b0bd9003d8977a5a" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:12:25.506251 30136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7f85d7037ce49a7b0bd9003d8977a5a" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:12:25.506283 30070 leader_election.cc:302] T c7f85d7037ce49a7b0bd9003d8977a5a P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:25.506506 30136 raft_consensus.cc:2454] T c7f85d7037ce49a7b0bd9003d8977a5a P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:25.506680 30266 raft_consensus.cc:2379] T c7f85d7037ce49a7b0bd9003d8977a5a P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a9094240860e43bfb129193f3bfd5be6 in current term 1: Already voted for candidate 1f10f80f13c84e80b03eda7085a3b012 in this term.
I20240503 02:12:25.506870 30135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7f85d7037ce49a7b0bd9003d8977a5a" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:12:25.507141 30135 raft_consensus.cc:3037] T c7f85d7037ce49a7b0bd9003d8977a5a P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:25.509454 30135 raft_consensus.cc:2454] T c7f85d7037ce49a7b0bd9003d8977a5a P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:25.509670 30869 raft_consensus.cc:2741] T c7f85d7037ce49a7b0bd9003d8977a5a P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240503 02:12:25.509923 30199 leader_election.cc:302] T c7f85d7037ce49a7b0bd9003d8977a5a P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, a9094240860e43bfb129193f3bfd5be6; no voters: 155735c7f8b14997ac4c770aee2a9401
I20240503 02:12:25.510195 30874 raft_consensus.cc:2781] T c7f85d7037ce49a7b0bd9003d8977a5a P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:25.510411 30874 raft_consensus.cc:686] T c7f85d7037ce49a7b0bd9003d8977a5a P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Becoming Leader. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Running, Role: LEADER
I20240503 02:12:25.510756 30874 consensus_queue.cc:237] T c7f85d7037ce49a7b0bd9003d8977a5a P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:25.513276 29734 catalog_manager.cc:5484] T c7f85d7037ce49a7b0bd9003d8977a5a P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4). New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:12:25.590986 30891 consensus_queue.cc:1035] T 006b2e20b1b04b18bee60fcd822949c7 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:25.592533 30891 consensus_queue.cc:1035] T 006b2e20b1b04b18bee60fcd822949c7 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:25.790764 30874 consensus_queue.cc:1035] T c5bb83b2a415451aa6909503a47662dd P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:25.796370 30874 consensus_queue.cc:1035] T c5bb83b2a415451aa6909503a47662dd P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:25.988595 30898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:25.988962 30898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:25.996766 30898 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:12:25.997257 30874 consensus_queue.cc:1035] T c7f85d7037ce49a7b0bd9003d8977a5a P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:26.001600 30006 raft_consensus.cc:3037] T c7f85d7037ce49a7b0bd9003d8977a5a P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:26.004915 30885 consensus_queue.cc:1035] T c7f85d7037ce49a7b0bd9003d8977a5a P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 247.969 ms
time per row: 0.123985 ms
I20240503 02:12:26.401640 29727 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:26.410168 29855 tablet_service.cc:1467] Processing CreateTablet for tablet e80b43971dc246d9bfd5e1c603f8415c (DEFAULT_TABLE table=rebalance_test_table_1 [id=28d7eeaed8114076b7afb44e4f89a2ea]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.410454 30116 tablet_service.cc:1467] Processing CreateTablet for tablet e80b43971dc246d9bfd5e1c603f8415c (DEFAULT_TABLE table=rebalance_test_table_1 [id=28d7eeaed8114076b7afb44e4f89a2ea]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.410684 29855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e80b43971dc246d9bfd5e1c603f8415c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.410610 30246 tablet_service.cc:1467] Processing CreateTablet for tablet e80b43971dc246d9bfd5e1c603f8415c (DEFAULT_TABLE table=rebalance_test_table_1 [id=28d7eeaed8114076b7afb44e4f89a2ea]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.410876 30116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e80b43971dc246d9bfd5e1c603f8415c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.411049 30246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e80b43971dc246d9bfd5e1c603f8415c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.414095 30115 tablet_service.cc:1467] Processing CreateTablet for tablet 96de1c26ae884ea18b63b328db040ba1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=28d7eeaed8114076b7afb44e4f89a2ea]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.414119 30245 tablet_service.cc:1467] Processing CreateTablet for tablet 96de1c26ae884ea18b63b328db040ba1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=28d7eeaed8114076b7afb44e4f89a2ea]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.414106 29985 tablet_service.cc:1467] Processing CreateTablet for tablet 46d608d84863424e8c65feac7c2e835f (DEFAULT_TABLE table=rebalance_test_table_1 [id=28d7eeaed8114076b7afb44e4f89a2ea]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.414453 30115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96de1c26ae884ea18b63b328db040ba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.414480 30245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96de1c26ae884ea18b63b328db040ba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.414525 29985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d608d84863424e8c65feac7c2e835f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.417632 30114 tablet_service.cc:1467] Processing CreateTablet for tablet 46d608d84863424e8c65feac7c2e835f (DEFAULT_TABLE table=rebalance_test_table_1 [id=28d7eeaed8114076b7afb44e4f89a2ea]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.418124 30114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d608d84863424e8c65feac7c2e835f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.418313 29854 tablet_service.cc:1467] Processing CreateTablet for tablet 96de1c26ae884ea18b63b328db040ba1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=28d7eeaed8114076b7afb44e4f89a2ea]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.418573 29853 tablet_service.cc:1467] Processing CreateTablet for tablet 46d608d84863424e8c65feac7c2e835f (DEFAULT_TABLE table=rebalance_test_table_1 [id=28d7eeaed8114076b7afb44e4f89a2ea]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:26.418740 29854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96de1c26ae884ea18b63b328db040ba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.419034 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d608d84863424e8c65feac7c2e835f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.431154 30962 tablet_bootstrap.cc:492] T 46d608d84863424e8c65feac7c2e835f P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:26.434199 30962 tablet_bootstrap.cc:654] T 46d608d84863424e8c65feac7c2e835f P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:26.434494 30961 tablet_bootstrap.cc:492] T e80b43971dc246d9bfd5e1c603f8415c P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:26.437305 30961 tablet_bootstrap.cc:654] T e80b43971dc246d9bfd5e1c603f8415c P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:26.440325 30963 tablet_bootstrap.cc:492] T 46d608d84863424e8c65feac7c2e835f P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:26.442243 30963 tablet_bootstrap.cc:654] T 46d608d84863424e8c65feac7c2e835f P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:26.443739 30962 tablet_bootstrap.cc:492] T 46d608d84863424e8c65feac7c2e835f P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:26.444010 30962 ts_tablet_manager.cc:1367] T 46d608d84863424e8c65feac7c2e835f P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.011s
I20240503 02:12:26.445504 30962 raft_consensus.cc:348] T 46d608d84863424e8c65feac7c2e835f P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:26.445842 30962 raft_consensus.cc:374] T 46d608d84863424e8c65feac7c2e835f P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:26.446022 30961 tablet_bootstrap.cc:492] T e80b43971dc246d9bfd5e1c603f8415c P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:26.446031 30962 raft_consensus.cc:729] T 46d608d84863424e8c65feac7c2e835f P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:26.446213 30961 ts_tablet_manager.cc:1367] T e80b43971dc246d9bfd5e1c603f8415c P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.006s
I20240503 02:12:26.446380 30962 consensus_queue.cc:260] T 46d608d84863424e8c65feac7c2e835f P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:26.447188 30962 ts_tablet_manager.cc:1398] T 46d608d84863424e8c65feac7c2e835f P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:26.447081 30961 raft_consensus.cc:348] T e80b43971dc246d9bfd5e1c603f8415c P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:26.447288 30961 raft_consensus.cc:374] T e80b43971dc246d9bfd5e1c603f8415c P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:26.447415 30961 raft_consensus.cc:729] T e80b43971dc246d9bfd5e1c603f8415c P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:26.447810 30961 consensus_queue.cc:260] T e80b43971dc246d9bfd5e1c603f8415c P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:26.448526 30961 ts_tablet_manager.cc:1398] T e80b43971dc246d9bfd5e1c603f8415c P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:12:26.448884 30961 tablet_bootstrap.cc:492] T 96de1c26ae884ea18b63b328db040ba1 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:26.451907 30961 tablet_bootstrap.cc:654] T 96de1c26ae884ea18b63b328db040ba1 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:26.453073 30960 tablet_bootstrap.cc:492] T e80b43971dc246d9bfd5e1c603f8415c P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:26.454154 30963 tablet_bootstrap.cc:492] T 46d608d84863424e8c65feac7c2e835f P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:26.454324 30963 ts_tablet_manager.cc:1367] T 46d608d84863424e8c65feac7c2e835f P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20240503 02:12:26.454953 30960 tablet_bootstrap.cc:654] T e80b43971dc246d9bfd5e1c603f8415c P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:26.455157 30963 raft_consensus.cc:348] T 46d608d84863424e8c65feac7c2e835f P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:26.455396 30963 raft_consensus.cc:374] T 46d608d84863424e8c65feac7c2e835f P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:26.455502 30963 raft_consensus.cc:729] T 46d608d84863424e8c65feac7c2e835f P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:26.455787 30963 consensus_queue.cc:260] T 46d608d84863424e8c65feac7c2e835f P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:26.456421 30963 ts_tablet_manager.cc:1398] T 46d608d84863424e8c65feac7c2e835f P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:26.456691 30963 tablet_bootstrap.cc:492] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:26.458218 30963 tablet_bootstrap.cc:654] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:26.461388 30960 tablet_bootstrap.cc:492] T e80b43971dc246d9bfd5e1c603f8415c P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:26.461560 30960 ts_tablet_manager.cc:1367] T e80b43971dc246d9bfd5e1c603f8415c P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20240503 02:12:26.461635 30961 tablet_bootstrap.cc:492] T 96de1c26ae884ea18b63b328db040ba1 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:26.461803 30961 ts_tablet_manager.cc:1367] T 96de1c26ae884ea18b63b328db040ba1 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20240503 02:12:26.462265 30960 raft_consensus.cc:348] T e80b43971dc246d9bfd5e1c603f8415c P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:26.462433 30960 raft_consensus.cc:374] T e80b43971dc246d9bfd5e1c603f8415c P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:26.462540 30960 raft_consensus.cc:729] T e80b43971dc246d9bfd5e1c603f8415c P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:26.462525 30961 raft_consensus.cc:348] T 96de1c26ae884ea18b63b328db040ba1 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:26.462730 30961 raft_consensus.cc:374] T 96de1c26ae884ea18b63b328db040ba1 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:26.462795 30961 raft_consensus.cc:729] T 96de1c26ae884ea18b63b328db040ba1 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:26.462802 30960 consensus_queue.cc:260] T e80b43971dc246d9bfd5e1c603f8415c P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:26.463012 30961 consensus_queue.cc:260] T 96de1c26ae884ea18b63b328db040ba1 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:26.463354 30960 ts_tablet_manager.cc:1398] T e80b43971dc246d9bfd5e1c603f8415c P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:26.463444 30961 ts_tablet_manager.cc:1398] T 96de1c26ae884ea18b63b328db040ba1 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:26.463640 30960 tablet_bootstrap.cc:492] T 96de1c26ae884ea18b63b328db040ba1 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:26.466812 30960 tablet_bootstrap.cc:654] T 96de1c26ae884ea18b63b328db040ba1 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:26.466970 30963 tablet_bootstrap.cc:492] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:26.467162 30963 ts_tablet_manager.cc:1367] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20240503 02:12:26.467937 30963 raft_consensus.cc:348] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:26.468183 30963 raft_consensus.cc:374] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:26.468274 30963 raft_consensus.cc:729] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:26.468549 30963 consensus_queue.cc:260] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:26.469203 30963 ts_tablet_manager.cc:1398] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:26.469496 30963 tablet_bootstrap.cc:492] T 96de1c26ae884ea18b63b328db040ba1 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:26.471066 30963 tablet_bootstrap.cc:654] T 96de1c26ae884ea18b63b328db040ba1 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:26.471756 30960 tablet_bootstrap.cc:492] T 96de1c26ae884ea18b63b328db040ba1 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:26.471926 30960 ts_tablet_manager.cc:1367] T 96de1c26ae884ea18b63b328db040ba1 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20240503 02:12:26.472733 30960 raft_consensus.cc:348] T 96de1c26ae884ea18b63b328db040ba1 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:26.472973 30960 raft_consensus.cc:374] T 96de1c26ae884ea18b63b328db040ba1 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:26.473083 30960 raft_consensus.cc:729] T 96de1c26ae884ea18b63b328db040ba1 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:26.473402 30960 consensus_queue.cc:260] T 96de1c26ae884ea18b63b328db040ba1 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:26.473932 30960 ts_tablet_manager.cc:1398] T 96de1c26ae884ea18b63b328db040ba1 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:12:26.474263 30960 tablet_bootstrap.cc:492] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:26.475850 30960 tablet_bootstrap.cc:654] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:26.478252 30960 tablet_bootstrap.cc:492] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:26.478560 30963 tablet_bootstrap.cc:492] T 96de1c26ae884ea18b63b328db040ba1 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:26.478713 30963 ts_tablet_manager.cc:1367] T 96de1c26ae884ea18b63b328db040ba1 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240503 02:12:26.478730 30960 ts_tablet_manager.cc:1367] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:12:26.479321 30963 raft_consensus.cc:348] T 96de1c26ae884ea18b63b328db040ba1 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:26.479338 30960 raft_consensus.cc:348] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:26.479475 30963 raft_consensus.cc:374] T 96de1c26ae884ea18b63b328db040ba1 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:26.479526 30960 raft_consensus.cc:374] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:26.479560 30963 raft_consensus.cc:729] T 96de1c26ae884ea18b63b328db040ba1 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:26.479625 30960 raft_consensus.cc:729] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:26.479777 30963 consensus_queue.cc:260] T 96de1c26ae884ea18b63b328db040ba1 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:26.480190 30963 ts_tablet_manager.cc:1398] T 96de1c26ae884ea18b63b328db040ba1 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:12:26.479920 30960 consensus_queue.cc:260] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:26.481096 30960 ts_tablet_manager.cc:1398] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:26.701236 30949 raft_consensus.cc:482] T 96de1c26ae884ea18b63b328db040ba1 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:26.701489 30949 raft_consensus.cc:504] T 96de1c26ae884ea18b63b328db040ba1 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:26.702481 30949 leader_election.cc:288] T 96de1c26ae884ea18b63b328db040ba1 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:26.703009 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96de1c26ae884ea18b63b328db040ba1" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:26.703038 30136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96de1c26ae884ea18b63b328db040ba1" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:12:26.703224 29875 raft_consensus.cc:2454] T 96de1c26ae884ea18b63b328db040ba1 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:26.703322 30136 raft_consensus.cc:2454] T 96de1c26ae884ea18b63b328db040ba1 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:26.703709 30202 leader_election.cc:302] T 96de1c26ae884ea18b63b328db040ba1 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:26.703980 30949 raft_consensus.cc:2781] T 96de1c26ae884ea18b63b328db040ba1 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:26.704095 30949 raft_consensus.cc:482] T 96de1c26ae884ea18b63b328db040ba1 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:26.704195 30949 raft_consensus.cc:3037] T 96de1c26ae884ea18b63b328db040ba1 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:26.706382 30949 raft_consensus.cc:504] T 96de1c26ae884ea18b63b328db040ba1 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:26.706926 30949 leader_election.cc:288] T 96de1c26ae884ea18b63b328db040ba1 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Requested vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:26.707434 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96de1c26ae884ea18b63b328db040ba1" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:12:26.707473 30136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96de1c26ae884ea18b63b328db040ba1" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:12:26.707623 29875 raft_consensus.cc:3037] T 96de1c26ae884ea18b63b328db040ba1 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:26.707738 30136 raft_consensus.cc:3037] T 96de1c26ae884ea18b63b328db040ba1 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:26.709648 29875 raft_consensus.cc:2454] T 96de1c26ae884ea18b63b328db040ba1 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:26.710129 30202 leader_election.cc:302] T 96de1c26ae884ea18b63b328db040ba1 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:26.710425 30906 raft_consensus.cc:2781] T 96de1c26ae884ea18b63b328db040ba1 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:26.710413 30941 raft_consensus.cc:482] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:26.710548 30906 raft_consensus.cc:686] T 96de1c26ae884ea18b63b328db040ba1 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Becoming Leader. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Running, Role: LEADER
I20240503 02:12:26.710588 30941 raft_consensus.cc:504] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:26.710812 30906 consensus_queue.cc:237] T 96de1c26ae884ea18b63b328db040ba1 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:26.711292 30941 leader_election.cc:288] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:26.711673 30006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d608d84863424e8c65feac7c2e835f" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:12:26.711980 30006 raft_consensus.cc:2454] T 46d608d84863424e8c65feac7c2e835f P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:12:26.712592 29809 leader_election.cc:302] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429; no voters:
I20240503 02:12:26.713125 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d608d84863424e8c65feac7c2e835f" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:12:26.713380 30134 raft_consensus.cc:2454] T 46d608d84863424e8c65feac7c2e835f P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:12:26.713680 30941 raft_consensus.cc:2781] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:26.713812 30941 raft_consensus.cc:482] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:26.713919 30941 raft_consensus.cc:3037] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:26.717581 30941 raft_consensus.cc:504] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:26.717907 30136 raft_consensus.cc:2454] T 96de1c26ae884ea18b63b328db040ba1 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:26.718367 29729 catalog_manager.cc:5484] T 96de1c26ae884ea18b63b328db040ba1 P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4). New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:12:26.718928 30006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d608d84863424e8c65feac7c2e835f" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:12:26.719192 30006 raft_consensus.cc:3037] T 46d608d84863424e8c65feac7c2e835f P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:26.719801 30136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d608d84863424e8c65feac7c2e835f" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:12:26.720129 30941 leader_election.cc:288] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Requested vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:26.719981 30136 raft_consensus.cc:3037] T 46d608d84863424e8c65feac7c2e835f P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:26.722503 30006 raft_consensus.cc:2454] T 46d608d84863424e8c65feac7c2e835f P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:12:26.723037 29809 leader_election.cc:302] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429; no voters:
I20240503 02:12:26.723392 30941 raft_consensus.cc:2781] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:26.723552 30941 raft_consensus.cc:686] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Becoming Leader. State: Replica: 16440e365430405fb10366ab70e2b429, State: Running, Role: LEADER
I20240503 02:12:26.723701 30136 raft_consensus.cc:2454] T 46d608d84863424e8c65feac7c2e835f P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:12:26.723882 30941 consensus_queue.cc:237] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:26.726867 29729 catalog_manager.cc:5484] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16440e365430405fb10366ab70e2b429 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:12:26.746910 30869 raft_consensus.cc:482] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:26.747149 30869 raft_consensus.cc:504] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:26.748184 30869 leader_election.cc:288] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:12:26.748697 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e80b43971dc246d9bfd5e1c603f8415c" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:12:26.748970 30266 raft_consensus.cc:2454] T e80b43971dc246d9bfd5e1c603f8415c P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 0.
I20240503 02:12:26.749737 30070 leader_election.cc:302] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:26.750026 30869 raft_consensus.cc:2781] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:26.750159 30869 raft_consensus.cc:482] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:26.750285 30869 raft_consensus.cc:3037] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:26.752817 30869 raft_consensus.cc:504] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:26.753674 30869 leader_election.cc:288] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Requested vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:12:26.754377 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e80b43971dc246d9bfd5e1c603f8415c" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:12:26.754588 30266 raft_consensus.cc:3037] T e80b43971dc246d9bfd5e1c603f8415c P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:26.756999 30266 raft_consensus.cc:2454] T e80b43971dc246d9bfd5e1c603f8415c P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 1.
I20240503 02:12:26.757419 30070 leader_election.cc:302] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:26.757699 30869 raft_consensus.cc:2781] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:26.757901 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e80b43971dc246d9bfd5e1c603f8415c" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:26.758055 29874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e80b43971dc246d9bfd5e1c603f8415c" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:12:26.758244 29875 raft_consensus.cc:2454] T e80b43971dc246d9bfd5e1c603f8415c P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 0.
I20240503 02:12:26.758774 30869 raft_consensus.cc:686] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Becoming Leader. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Running, Role: LEADER
I20240503 02:12:26.759302 30869 consensus_queue.cc:237] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:26.762620 29729 catalog_manager.cc:5484] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9094240860e43bfb129193f3bfd5be6 (127.22.108.3). New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:12:27.112545 30941 consensus_queue.cc:1035] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:27.118923 30984 consensus_queue.cc:1035] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:27.143226 30979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:27.143530 30979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 02:12:27.148157 30869 consensus_queue.cc:1035] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:27.151763 30979 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:12:27.152827 29875 raft_consensus.cc:3037] T e80b43971dc246d9bfd5e1c603f8415c P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:27.155952 30869 consensus_queue.cc:1035] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:27.178229 30949 consensus_queue.cc:1035] T 96de1c26ae884ea18b63b328db040ba1 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:27.182456 30949 consensus_queue.cc:1035] T 96de1c26ae884ea18b63b328db040ba1 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 226.001 ms
time per row: 0.113001 ms
I20240503 02:12:27.534850 29729 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:27.543524 29985 tablet_service.cc:1467] Processing CreateTablet for tablet 70820e67bc3740a08bcf0510a055c544 (DEFAULT_TABLE table=rebalance_test_table_2 [id=2de6ddd95bd0493cac0dd361e1642e8f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.544054 29985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70820e67bc3740a08bcf0510a055c544. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.544018 30246 tablet_service.cc:1467] Processing CreateTablet for tablet 70820e67bc3740a08bcf0510a055c544 (DEFAULT_TABLE table=rebalance_test_table_2 [id=2de6ddd95bd0493cac0dd361e1642e8f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.544077 29855 tablet_service.cc:1467] Processing CreateTablet for tablet 70820e67bc3740a08bcf0510a055c544 (DEFAULT_TABLE table=rebalance_test_table_2 [id=2de6ddd95bd0493cac0dd361e1642e8f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.544457 30246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70820e67bc3740a08bcf0510a055c544. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.544538 29855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70820e67bc3740a08bcf0510a055c544. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.545447 30114 tablet_service.cc:1467] Processing CreateTablet for tablet 8706c9b491b54500963bb81e6e459ddd (DEFAULT_TABLE table=rebalance_test_table_2 [id=2de6ddd95bd0493cac0dd361e1642e8f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.545861 30114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8706c9b491b54500963bb81e6e459ddd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.550158 29853 tablet_service.cc:1467] Processing CreateTablet for tablet bd91cf40fd654426a11789f37c4dca47 (DEFAULT_TABLE table=rebalance_test_table_2 [id=2de6ddd95bd0493cac0dd361e1642e8f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.550284 29984 tablet_service.cc:1467] Processing CreateTablet for tablet 8706c9b491b54500963bb81e6e459ddd (DEFAULT_TABLE table=rebalance_test_table_2 [id=2de6ddd95bd0493cac0dd361e1642e8f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.550622 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd91cf40fd654426a11789f37c4dca47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.550662 29984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8706c9b491b54500963bb81e6e459ddd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.553174 29986 tablet_service.cc:1467] Processing CreateTablet for tablet bd91cf40fd654426a11789f37c4dca47 (DEFAULT_TABLE table=rebalance_test_table_2 [id=2de6ddd95bd0493cac0dd361e1642e8f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.553645 29986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd91cf40fd654426a11789f37c4dca47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.553539 30245 tablet_service.cc:1467] Processing CreateTablet for tablet 8706c9b491b54500963bb81e6e459ddd (DEFAULT_TABLE table=rebalance_test_table_2 [id=2de6ddd95bd0493cac0dd361e1642e8f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.554010 30245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8706c9b491b54500963bb81e6e459ddd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.557641 30244 tablet_service.cc:1467] Processing CreateTablet for tablet bd91cf40fd654426a11789f37c4dca47 (DEFAULT_TABLE table=rebalance_test_table_2 [id=2de6ddd95bd0493cac0dd361e1642e8f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.558154 30244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd91cf40fd654426a11789f37c4dca47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.577131 31044 tablet_bootstrap.cc:492] T 8706c9b491b54500963bb81e6e459ddd P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:27.579774 31045 tablet_bootstrap.cc:492] T 70820e67bc3740a08bcf0510a055c544 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:27.580691 31044 tablet_bootstrap.cc:654] T 8706c9b491b54500963bb81e6e459ddd P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:27.581558 31045 tablet_bootstrap.cc:654] T 70820e67bc3740a08bcf0510a055c544 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:27.584664 31043 tablet_bootstrap.cc:492] T 70820e67bc3740a08bcf0510a055c544 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:27.584687 31046 tablet_bootstrap.cc:492] T 70820e67bc3740a08bcf0510a055c544 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:27.586262 31046 tablet_bootstrap.cc:654] T 70820e67bc3740a08bcf0510a055c544 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:27.586282 31043 tablet_bootstrap.cc:654] T 70820e67bc3740a08bcf0510a055c544 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:27.586300 31044 tablet_bootstrap.cc:492] T 8706c9b491b54500963bb81e6e459ddd P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:27.586434 31044 ts_tablet_manager.cc:1367] T 8706c9b491b54500963bb81e6e459ddd P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.002s
I20240503 02:12:27.587359 31044 raft_consensus.cc:348] T 8706c9b491b54500963bb81e6e459ddd P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:27.587793 31044 raft_consensus.cc:374] T 8706c9b491b54500963bb81e6e459ddd P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:27.587949 31044 raft_consensus.cc:729] T 8706c9b491b54500963bb81e6e459ddd P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:27.588068 31045 tablet_bootstrap.cc:492] T 70820e67bc3740a08bcf0510a055c544 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:27.588263 31045 ts_tablet_manager.cc:1367] T 70820e67bc3740a08bcf0510a055c544 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240503 02:12:27.588577 31043 tablet_bootstrap.cc:492] T 70820e67bc3740a08bcf0510a055c544 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:27.588692 31046 tablet_bootstrap.cc:492] T 70820e67bc3740a08bcf0510a055c544 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:27.588737 31043 ts_tablet_manager.cc:1367] T 70820e67bc3740a08bcf0510a055c544 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 02:12:27.588864 31046 ts_tablet_manager.cc:1367] T 70820e67bc3740a08bcf0510a055c544 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:12:27.589156 31045 raft_consensus.cc:348] T 70820e67bc3740a08bcf0510a055c544 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:27.589395 31045 raft_consensus.cc:374] T 70820e67bc3740a08bcf0510a055c544 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:27.589527 31045 raft_consensus.cc:729] T 70820e67bc3740a08bcf0510a055c544 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:27.589468 31044 consensus_queue.cc:260] T 8706c9b491b54500963bb81e6e459ddd P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:27.589742 31043 raft_consensus.cc:348] T 70820e67bc3740a08bcf0510a055c544 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:27.589809 31045 consensus_queue.cc:260] T 70820e67bc3740a08bcf0510a055c544 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:27.590013 31043 raft_consensus.cc:374] T 70820e67bc3740a08bcf0510a055c544 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:27.590170 31043 raft_consensus.cc:729] T 70820e67bc3740a08bcf0510a055c544 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:27.590436 31045 ts_tablet_manager.cc:1398] T 70820e67bc3740a08bcf0510a055c544 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:27.590797 31045 tablet_bootstrap.cc:492] T bd91cf40fd654426a11789f37c4dca47 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:27.590477 31043 consensus_queue.cc:260] T 70820e67bc3740a08bcf0510a055c544 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:27.591358 31043 ts_tablet_manager.cc:1398] T 70820e67bc3740a08bcf0510a055c544 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:27.591710 31043 tablet_bootstrap.cc:492] T 8706c9b491b54500963bb81e6e459ddd P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:27.592221 31046 raft_consensus.cc:348] T 70820e67bc3740a08bcf0510a055c544 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:27.592512 31045 tablet_bootstrap.cc:654] T bd91cf40fd654426a11789f37c4dca47 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:27.593187 31044 ts_tablet_manager.cc:1398] T 8706c9b491b54500963bb81e6e459ddd P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20240503 02:12:27.593480 31046 raft_consensus.cc:374] T 70820e67bc3740a08bcf0510a055c544 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:27.593626 31046 raft_consensus.cc:729] T 70820e67bc3740a08bcf0510a055c544 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:27.593710 31043 tablet_bootstrap.cc:654] T 8706c9b491b54500963bb81e6e459ddd P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:27.593859 31046 consensus_queue.cc:260] T 70820e67bc3740a08bcf0510a055c544 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:27.594460 31046 ts_tablet_manager.cc:1398] T 70820e67bc3740a08bcf0510a055c544 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 02:12:27.594763 31046 tablet_bootstrap.cc:492] T 8706c9b491b54500963bb81e6e459ddd P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:27.596343 31046 tablet_bootstrap.cc:654] T 8706c9b491b54500963bb81e6e459ddd P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:27.602706 31045 tablet_bootstrap.cc:492] T bd91cf40fd654426a11789f37c4dca47 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:27.602690 31043 tablet_bootstrap.cc:492] T 8706c9b491b54500963bb81e6e459ddd P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:27.602872 31043 ts_tablet_manager.cc:1367] T 8706c9b491b54500963bb81e6e459ddd P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20240503 02:12:27.602874 31045 ts_tablet_manager.cc:1367] T bd91cf40fd654426a11789f37c4dca47 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20240503 02:12:27.602986 31046 tablet_bootstrap.cc:492] T 8706c9b491b54500963bb81e6e459ddd P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:27.603143 31046 ts_tablet_manager.cc:1367] T 8706c9b491b54500963bb81e6e459ddd P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20240503 02:12:27.603600 31043 raft_consensus.cc:348] T 8706c9b491b54500963bb81e6e459ddd P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:27.603613 31045 raft_consensus.cc:348] T bd91cf40fd654426a11789f37c4dca47 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:27.603724 31046 raft_consensus.cc:348] T 8706c9b491b54500963bb81e6e459ddd P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:27.603857 31045 raft_consensus.cc:374] T bd91cf40fd654426a11789f37c4dca47 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:27.603857 31043 raft_consensus.cc:374] T 8706c9b491b54500963bb81e6e459ddd P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:27.603937 31046 raft_consensus.cc:374] T 8706c9b491b54500963bb81e6e459ddd P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:27.603937 31043 raft_consensus.cc:729] T 8706c9b491b54500963bb81e6e459ddd P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:27.603991 31045 raft_consensus.cc:729] T bd91cf40fd654426a11789f37c4dca47 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:27.604028 31046 raft_consensus.cc:729] T 8706c9b491b54500963bb81e6e459ddd P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:27.604200 31043 consensus_queue.cc:260] T 8706c9b491b54500963bb81e6e459ddd P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:27.604288 31046 consensus_queue.cc:260] T 8706c9b491b54500963bb81e6e459ddd P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:27.604274 31045 consensus_queue.cc:260] T bd91cf40fd654426a11789f37c4dca47 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:27.604776 31043 ts_tablet_manager.cc:1398] T 8706c9b491b54500963bb81e6e459ddd P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:27.604781 31046 ts_tablet_manager.cc:1398] T 8706c9b491b54500963bb81e6e459ddd P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:27.605015 31045 ts_tablet_manager.cc:1398] T bd91cf40fd654426a11789f37c4dca47 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 02:12:27.605067 31046 tablet_bootstrap.cc:492] T bd91cf40fd654426a11789f37c4dca47 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:27.605209 31043 tablet_bootstrap.cc:492] T bd91cf40fd654426a11789f37c4dca47 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:27.606628 31046 tablet_bootstrap.cc:654] T bd91cf40fd654426a11789f37c4dca47 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:27.606659 31043 tablet_bootstrap.cc:654] T bd91cf40fd654426a11789f37c4dca47 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:27.613258 31046 tablet_bootstrap.cc:492] T bd91cf40fd654426a11789f37c4dca47 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:27.613401 31046 ts_tablet_manager.cc:1367] T bd91cf40fd654426a11789f37c4dca47 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20240503 02:12:27.613577 31043 tablet_bootstrap.cc:492] T bd91cf40fd654426a11789f37c4dca47 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:27.613713 31043 ts_tablet_manager.cc:1367] T bd91cf40fd654426a11789f37c4dca47 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.001s
I20240503 02:12:27.614197 30949 raft_consensus.cc:482] T 70820e67bc3740a08bcf0510a055c544 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:27.614167 31046 raft_consensus.cc:348] T bd91cf40fd654426a11789f37c4dca47 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:27.614331 31046 raft_consensus.cc:374] T bd91cf40fd654426a11789f37c4dca47 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:27.614444 31046 raft_consensus.cc:729] T bd91cf40fd654426a11789f37c4dca47 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:27.614356 30949 raft_consensus.cc:504] T 70820e67bc3740a08bcf0510a055c544 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:27.614459 31043 raft_consensus.cc:348] T bd91cf40fd654426a11789f37c4dca47 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:27.614682 31043 raft_consensus.cc:374] T bd91cf40fd654426a11789f37c4dca47 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:27.614666 31046 consensus_queue.cc:260] T bd91cf40fd654426a11789f37c4dca47 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:27.614786 31043 raft_consensus.cc:729] T bd91cf40fd654426a11789f37c4dca47 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:27.614993 31043 consensus_queue.cc:260] T bd91cf40fd654426a11789f37c4dca47 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:27.615104 30949 leader_election.cc:288] T 70820e67bc3740a08bcf0510a055c544 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:27.615276 31046 ts_tablet_manager.cc:1398] T bd91cf40fd654426a11789f37c4dca47 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:27.615592 31043 ts_tablet_manager.cc:1398] T bd91cf40fd654426a11789f37c4dca47 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:27.615808 30006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70820e67bc3740a08bcf0510a055c544" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:12:27.615983 30006 raft_consensus.cc:2454] T 70820e67bc3740a08bcf0510a055c544 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:27.616190 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70820e67bc3740a08bcf0510a055c544" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:27.616366 30200 leader_election.cc:302] T 70820e67bc3740a08bcf0510a055c544 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:27.616448 29875 raft_consensus.cc:2454] T 70820e67bc3740a08bcf0510a055c544 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:27.616654 30949 raft_consensus.cc:2781] T 70820e67bc3740a08bcf0510a055c544 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:27.616786 30949 raft_consensus.cc:482] T 70820e67bc3740a08bcf0510a055c544 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:27.616871 30949 raft_consensus.cc:3037] T 70820e67bc3740a08bcf0510a055c544 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:27.619334 30949 raft_consensus.cc:504] T 70820e67bc3740a08bcf0510a055c544 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:27.619889 30949 leader_election.cc:288] T 70820e67bc3740a08bcf0510a055c544 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Requested vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:27.620272 30006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70820e67bc3740a08bcf0510a055c544" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:12:27.620354 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70820e67bc3740a08bcf0510a055c544" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:12:27.620462 30006 raft_consensus.cc:3037] T 70820e67bc3740a08bcf0510a055c544 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:27.620556 29875 raft_consensus.cc:3037] T 70820e67bc3740a08bcf0510a055c544 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:27.622292 30006 raft_consensus.cc:2454] T 70820e67bc3740a08bcf0510a055c544 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:27.622429 29875 raft_consensus.cc:2454] T 70820e67bc3740a08bcf0510a055c544 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:27.622686 30200 leader_election.cc:302] T 70820e67bc3740a08bcf0510a055c544 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:27.622969 30949 raft_consensus.cc:2781] T 70820e67bc3740a08bcf0510a055c544 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:27.623112 30949 raft_consensus.cc:686] T 70820e67bc3740a08bcf0510a055c544 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Becoming Leader. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Running, Role: LEADER
I20240503 02:12:27.623401 30949 consensus_queue.cc:237] T 70820e67bc3740a08bcf0510a055c544 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:27.625806 29733 catalog_manager.cc:5484] T 70820e67bc3740a08bcf0510a055c544 P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4). New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:12:27.634958 30980 raft_consensus.cc:482] T bd91cf40fd654426a11789f37c4dca47 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:27.635123 30980 raft_consensus.cc:504] T bd91cf40fd654426a11789f37c4dca47 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:27.635856 30980 leader_election.cc:288] T bd91cf40fd654426a11789f37c4dca47 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:27.636487 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd91cf40fd654426a11789f37c4dca47" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:27.636773 29875 raft_consensus.cc:2454] T bd91cf40fd654426a11789f37c4dca47 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:12:27.637217 29942 leader_election.cc:302] T bd91cf40fd654426a11789f37c4dca47 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429; no voters:
I20240503 02:12:27.637493 30980 raft_consensus.cc:2781] T bd91cf40fd654426a11789f37c4dca47 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:27.637609 30980 raft_consensus.cc:482] T bd91cf40fd654426a11789f37c4dca47 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:27.637724 30980 raft_consensus.cc:3037] T bd91cf40fd654426a11789f37c4dca47 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:27.640201 30980 raft_consensus.cc:504] T bd91cf40fd654426a11789f37c4dca47 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:27.640926 30980 leader_election.cc:288] T bd91cf40fd654426a11789f37c4dca47 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Requested vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:27.641423 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd91cf40fd654426a11789f37c4dca47" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:12:27.641678 29875 raft_consensus.cc:3037] T bd91cf40fd654426a11789f37c4dca47 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:27.642897 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd91cf40fd654426a11789f37c4dca47" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:12:27.643170 30266 raft_consensus.cc:2454] T bd91cf40fd654426a11789f37c4dca47 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:12:27.643059 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd91cf40fd654426a11789f37c4dca47" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:12:27.643378 30265 raft_consensus.cc:3037] T bd91cf40fd654426a11789f37c4dca47 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:27.644029 29875 raft_consensus.cc:2454] T bd91cf40fd654426a11789f37c4dca47 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:12:27.644385 29942 leader_election.cc:302] T bd91cf40fd654426a11789f37c4dca47 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429; no voters:
I20240503 02:12:27.644694 30980 raft_consensus.cc:2781] T bd91cf40fd654426a11789f37c4dca47 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:27.645259 30980 raft_consensus.cc:686] T bd91cf40fd654426a11789f37c4dca47 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Becoming Leader. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Running, Role: LEADER
I20240503 02:12:27.645772 30980 consensus_queue.cc:237] T bd91cf40fd654426a11789f37c4dca47 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:27.646132 30265 raft_consensus.cc:2454] T bd91cf40fd654426a11789f37c4dca47 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:12:27.649060 29733 catalog_manager.cc:5484] T bd91cf40fd654426a11789f37c4dca47 P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: term changed from 0 to 1, leader changed from <none> to 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2). New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:12:27.765153 30869 raft_consensus.cc:482] T 8706c9b491b54500963bb81e6e459ddd P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:27.765340 30869 raft_consensus.cc:504] T 8706c9b491b54500963bb81e6e459ddd P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:27.765859 30869 leader_election.cc:288] T 8706c9b491b54500963bb81e6e459ddd P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:12:27.766381 30006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8706c9b491b54500963bb81e6e459ddd" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:12:27.766458 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8706c9b491b54500963bb81e6e459ddd" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:12:27.766680 30006 raft_consensus.cc:2454] T 8706c9b491b54500963bb81e6e459ddd P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 0.
I20240503 02:12:27.766681 30265 raft_consensus.cc:2454] T 8706c9b491b54500963bb81e6e459ddd P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 0.
I20240503 02:12:27.767097 30070 leader_election.cc:302] T 8706c9b491b54500963bb81e6e459ddd P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:27.767379 30869 raft_consensus.cc:2781] T 8706c9b491b54500963bb81e6e459ddd P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:27.767462 30869 raft_consensus.cc:482] T 8706c9b491b54500963bb81e6e459ddd P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:27.767565 30869 raft_consensus.cc:3037] T 8706c9b491b54500963bb81e6e459ddd P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:27.769433 30869 raft_consensus.cc:504] T 8706c9b491b54500963bb81e6e459ddd P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:27.769953 30869 leader_election.cc:288] T 8706c9b491b54500963bb81e6e459ddd P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Requested vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:12:27.770363 30006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8706c9b491b54500963bb81e6e459ddd" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:12:27.770408 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8706c9b491b54500963bb81e6e459ddd" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:12:27.770541 30006 raft_consensus.cc:3037] T 8706c9b491b54500963bb81e6e459ddd P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:27.770658 30265 raft_consensus.cc:3037] T 8706c9b491b54500963bb81e6e459ddd P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:27.772365 30006 raft_consensus.cc:2454] T 8706c9b491b54500963bb81e6e459ddd P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 1.
I20240503 02:12:27.772426 30265 raft_consensus.cc:2454] T 8706c9b491b54500963bb81e6e459ddd P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 1.
I20240503 02:12:27.772773 30070 leader_election.cc:302] T 8706c9b491b54500963bb81e6e459ddd P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:27.773066 30869 raft_consensus.cc:2781] T 8706c9b491b54500963bb81e6e459ddd P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:27.773237 30869 raft_consensus.cc:686] T 8706c9b491b54500963bb81e6e459ddd P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Becoming Leader. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Running, Role: LEADER
I20240503 02:12:27.773492 30869 consensus_queue.cc:237] T 8706c9b491b54500963bb81e6e459ddd P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:27.775904 29733 catalog_manager.cc:5484] T 8706c9b491b54500963bb81e6e459ddd P a9094240860e43bfb129193f3bfd5be6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9094240860e43bfb129193f3bfd5be6 (127.22.108.3). New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:12:28.063694 30949 consensus_queue.cc:1035] T 70820e67bc3740a08bcf0510a055c544 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:28.076129 30949 consensus_queue.cc:1035] T 70820e67bc3740a08bcf0510a055c544 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:28.100605 30980 consensus_queue.cc:1035] T bd91cf40fd654426a11789f37c4dca47 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:28.106045 30980 consensus_queue.cc:1035] T bd91cf40fd654426a11789f37c4dca47 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:28.140952 31061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:28.141331 31061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:28.149335 31061 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 02:12:28.233053 30869 consensus_queue.cc:1035] T 8706c9b491b54500963bb81e6e459ddd P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:28.251305 30977 consensus_queue.cc:1035] T 8706c9b491b54500963bb81e6e459ddd P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:28.517683 29942 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.1:36719, user_credentials={real_user=slave}} blocked reactor thread for 52263us
INSERT report
rows total: 2000
time total: 358.28 ms
time per row: 0.17914 ms
I20240503 02:12:28.659957 29732 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:28.668466 29855 tablet_service.cc:1467] Processing CreateTablet for tablet 35a5a7f4fe184b9fadc94c98b61deaf0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3131069dba4a4f9aa26d4c134fcb4704]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.668725 30114 tablet_service.cc:1467] Processing CreateTablet for tablet 35a5a7f4fe184b9fadc94c98b61deaf0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3131069dba4a4f9aa26d4c134fcb4704]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.668938 29855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35a5a7f4fe184b9fadc94c98b61deaf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.669067 29985 tablet_service.cc:1467] Processing CreateTablet for tablet 35a5a7f4fe184b9fadc94c98b61deaf0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3131069dba4a4f9aa26d4c134fcb4704]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.669499 29985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35a5a7f4fe184b9fadc94c98b61deaf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.669662 30114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35a5a7f4fe184b9fadc94c98b61deaf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.669906 30115 tablet_service.cc:1467] Processing CreateTablet for tablet 683d4041910f4355910e188c0b7df6c7 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3131069dba4a4f9aa26d4c134fcb4704]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.670253 30115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 683d4041910f4355910e188c0b7df6c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.672648 30116 tablet_service.cc:1467] Processing CreateTablet for tablet 7527cc5931e24fa180efedbf81438efd (DEFAULT_TABLE table=rebalance_test_table_3 [id=3131069dba4a4f9aa26d4c134fcb4704]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.672968 30116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7527cc5931e24fa180efedbf81438efd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.678050 30246 tablet_service.cc:1467] Processing CreateTablet for tablet 7527cc5931e24fa180efedbf81438efd (DEFAULT_TABLE table=rebalance_test_table_3 [id=3131069dba4a4f9aa26d4c134fcb4704]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.678439 30246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7527cc5931e24fa180efedbf81438efd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.681335 30245 tablet_service.cc:1467] Processing CreateTablet for tablet 683d4041910f4355910e188c0b7df6c7 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3131069dba4a4f9aa26d4c134fcb4704]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.681670 30245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 683d4041910f4355910e188c0b7df6c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.684305 31128 tablet_bootstrap.cc:492] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:28.685192 29853 tablet_service.cc:1467] Processing CreateTablet for tablet 7527cc5931e24fa180efedbf81438efd (DEFAULT_TABLE table=rebalance_test_table_3 [id=3131069dba4a4f9aa26d4c134fcb4704]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.685580 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7527cc5931e24fa180efedbf81438efd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.685483 29986 tablet_service.cc:1467] Processing CreateTablet for tablet 683d4041910f4355910e188c0b7df6c7 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3131069dba4a4f9aa26d4c134fcb4704]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.685915 29986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 683d4041910f4355910e188c0b7df6c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.693781 31131 tablet_bootstrap.cc:492] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:28.694366 31128 tablet_bootstrap.cc:654] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:28.695941 31131 tablet_bootstrap.cc:654] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:28.697314 31129 tablet_bootstrap.cc:492] T 35a5a7f4fe184b9fadc94c98b61deaf0 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:28.699223 31129 tablet_bootstrap.cc:654] T 35a5a7f4fe184b9fadc94c98b61deaf0 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:28.702562 31130 tablet_bootstrap.cc:492] T 683d4041910f4355910e188c0b7df6c7 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:28.706140 31131 tablet_bootstrap.cc:492] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:28.706410 31131 ts_tablet_manager.cc:1367] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20240503 02:12:28.707343 31129 tablet_bootstrap.cc:492] T 35a5a7f4fe184b9fadc94c98b61deaf0 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:28.707654 31129 ts_tablet_manager.cc:1367] T 35a5a7f4fe184b9fadc94c98b61deaf0 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.007s
I20240503 02:12:28.707509 31131 raft_consensus.cc:348] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:28.707932 31131 raft_consensus.cc:374] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:28.708097 31131 raft_consensus.cc:729] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:28.708405 31129 raft_consensus.cc:348] T 35a5a7f4fe184b9fadc94c98b61deaf0 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:28.708670 31129 raft_consensus.cc:374] T 35a5a7f4fe184b9fadc94c98b61deaf0 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:28.708519 31131 consensus_queue.cc:260] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:28.708796 31129 raft_consensus.cc:729] T 35a5a7f4fe184b9fadc94c98b61deaf0 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:28.709097 31129 consensus_queue.cc:260] T 35a5a7f4fe184b9fadc94c98b61deaf0 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:28.709295 31131 ts_tablet_manager.cc:1398] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:28.709658 31129 ts_tablet_manager.cc:1398] T 35a5a7f4fe184b9fadc94c98b61deaf0 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:28.709767 31131 tablet_bootstrap.cc:492] T 683d4041910f4355910e188c0b7df6c7 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:28.709955 31129 tablet_bootstrap.cc:492] T 683d4041910f4355910e188c0b7df6c7 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:28.711292 31131 tablet_bootstrap.cc:654] T 683d4041910f4355910e188c0b7df6c7 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:28.711341 31129 tablet_bootstrap.cc:654] T 683d4041910f4355910e188c0b7df6c7 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:28.719244 31129 tablet_bootstrap.cc:492] T 683d4041910f4355910e188c0b7df6c7 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:28.720417 31129 ts_tablet_manager.cc:1367] T 683d4041910f4355910e188c0b7df6c7 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20240503 02:12:28.721351 31129 raft_consensus.cc:348] T 683d4041910f4355910e188c0b7df6c7 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:28.721650 31129 raft_consensus.cc:374] T 683d4041910f4355910e188c0b7df6c7 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:28.721782 31129 raft_consensus.cc:729] T 683d4041910f4355910e188c0b7df6c7 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:28.722131 31129 consensus_queue.cc:260] T 683d4041910f4355910e188c0b7df6c7 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:28.722628 31130 tablet_bootstrap.cc:654] T 683d4041910f4355910e188c0b7df6c7 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:28.722954 31129 ts_tablet_manager.cc:1398] T 683d4041910f4355910e188c0b7df6c7 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:12:28.723160 31128 tablet_bootstrap.cc:492] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:28.723352 31129 tablet_bootstrap.cc:492] T 7527cc5931e24fa180efedbf81438efd P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:28.723347 31128 ts_tablet_manager.cc:1367] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.039s user 0.013s sys 0.007s
I20240503 02:12:28.724298 31128 raft_consensus.cc:348] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:28.724561 31128 raft_consensus.cc:374] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:28.724699 31128 raft_consensus.cc:729] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:28.724992 31129 tablet_bootstrap.cc:654] T 7527cc5931e24fa180efedbf81438efd P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:28.725003 31128 consensus_queue.cc:260] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:28.725783 31128 ts_tablet_manager.cc:1398] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:12:28.726029 31119 raft_consensus.cc:482] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:28.726167 31128 tablet_bootstrap.cc:492] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:28.726183 31119 raft_consensus.cc:504] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:28.727037 31119 leader_election.cc:288] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:28.727418 31130 tablet_bootstrap.cc:492] T 683d4041910f4355910e188c0b7df6c7 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:28.727574 31130 ts_tablet_manager.cc:1367] T 683d4041910f4355910e188c0b7df6c7 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.002s
I20240503 02:12:28.728098 31128 tablet_bootstrap.cc:654] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:28.719220 31131 tablet_bootstrap.cc:492] T 683d4041910f4355910e188c0b7df6c7 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:28.728443 31131 ts_tablet_manager.cc:1367] T 683d4041910f4355910e188c0b7df6c7 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.002s
I20240503 02:12:28.728436 31130 raft_consensus.cc:348] T 683d4041910f4355910e188c0b7df6c7 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:28.728513 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35a5a7f4fe184b9fadc94c98b61deaf0" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:28.728659 31130 raft_consensus.cc:374] T 683d4041910f4355910e188c0b7df6c7 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:28.728731 29875 raft_consensus.cc:2454] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:12:28.728766 31130 raft_consensus.cc:729] T 683d4041910f4355910e188c0b7df6c7 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:28.729018 31130 consensus_queue.cc:260] T 683d4041910f4355910e188c0b7df6c7 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:28.729270 29942 leader_election.cc:302] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429; no voters:
I20240503 02:12:28.729300 31131 raft_consensus.cc:348] T 683d4041910f4355910e188c0b7df6c7 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:28.729523 31131 raft_consensus.cc:374] T 683d4041910f4355910e188c0b7df6c7 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:28.729612 31130 ts_tablet_manager.cc:1398] T 683d4041910f4355910e188c0b7df6c7 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:28.729610 31131 raft_consensus.cc:729] T 683d4041910f4355910e188c0b7df6c7 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:28.729911 31130 tablet_bootstrap.cc:492] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:28.729836 31131 consensus_queue.cc:260] T 683d4041910f4355910e188c0b7df6c7 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:28.730309 31120 raft_consensus.cc:2781] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:28.730351 31131 ts_tablet_manager.cc:1398] T 683d4041910f4355910e188c0b7df6c7 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:28.730417 31120 raft_consensus.cc:482] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:28.730577 31120 raft_consensus.cc:3037] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:28.730919 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35a5a7f4fe184b9fadc94c98b61deaf0" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:12:28.731117 30134 raft_consensus.cc:2454] T 35a5a7f4fe184b9fadc94c98b61deaf0 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:12:28.731382 31129 tablet_bootstrap.cc:492] T 7527cc5931e24fa180efedbf81438efd P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:28.731519 31129 ts_tablet_manager.cc:1367] T 7527cc5931e24fa180efedbf81438efd P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20240503 02:12:28.732234 31129 raft_consensus.cc:348] T 7527cc5931e24fa180efedbf81438efd P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:28.732414 31129 raft_consensus.cc:374] T 7527cc5931e24fa180efedbf81438efd P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:28.732513 31129 raft_consensus.cc:729] T 7527cc5931e24fa180efedbf81438efd P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:28.732656 31130 tablet_bootstrap.cc:654] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:28.732771 31129 consensus_queue.cc:260] T 7527cc5931e24fa180efedbf81438efd P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:28.733422 31129 ts_tablet_manager.cc:1398] T 7527cc5931e24fa180efedbf81438efd P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:12:28.734176 31120 raft_consensus.cc:504] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:28.735018 31120 leader_election.cc:288] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Requested vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:28.735896 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35a5a7f4fe184b9fadc94c98b61deaf0" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:12:28.736083 29875 raft_consensus.cc:3037] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:28.738986 29875 raft_consensus.cc:2454] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:12:28.739380 29942 leader_election.cc:302] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429; no voters:
I20240503 02:12:28.739682 31120 raft_consensus.cc:2781] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:28.739847 31120 raft_consensus.cc:686] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Becoming Leader. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Running, Role: LEADER
I20240503 02:12:28.740077 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35a5a7f4fe184b9fadc94c98b61deaf0" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:12:28.740098 31120 consensus_queue.cc:237] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:28.740249 30134 raft_consensus.cc:3037] T 35a5a7f4fe184b9fadc94c98b61deaf0 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:28.743217 31130 tablet_bootstrap.cc:492] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:28.743438 31130 ts_tablet_manager.cc:1367] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.007s
I20240503 02:12:28.743567 30134 raft_consensus.cc:2454] T 35a5a7f4fe184b9fadc94c98b61deaf0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:12:28.744542 31130 raft_consensus.cc:348] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:28.744841 31130 raft_consensus.cc:374] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:28.744984 31130 raft_consensus.cc:729] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:28.745337 31130 consensus_queue.cc:260] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:28.746225 31130 ts_tablet_manager.cc:1398] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:12:28.747951 31128 tablet_bootstrap.cc:492] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:28.748128 31128 ts_tablet_manager.cc:1367] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.010s
I20240503 02:12:28.748670 29732 catalog_manager.cc:5484] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: term changed from 0 to 1, leader changed from <none> to 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2). New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:12:28.748909 31128 raft_consensus.cc:348] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:28.749331 31128 raft_consensus.cc:374] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:28.749462 31128 raft_consensus.cc:729] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:28.749742 31128 consensus_queue.cc:260] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:28.750344 31128 ts_tablet_manager.cc:1398] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:28.838346 31065 raft_consensus.cc:482] T 683d4041910f4355910e188c0b7df6c7 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:28.838510 31065 raft_consensus.cc:504] T 683d4041910f4355910e188c0b7df6c7 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:28.839160 31065 leader_election.cc:288] T 683d4041910f4355910e188c0b7df6c7 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:12:28.839706 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "683d4041910f4355910e188c0b7df6c7" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:12:28.839740 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "683d4041910f4355910e188c0b7df6c7" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:12:28.839939 30005 raft_consensus.cc:2454] T 683d4041910f4355910e188c0b7df6c7 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:28.839929 30134 raft_consensus.cc:2454] T 683d4041910f4355910e188c0b7df6c7 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:28.840339 30199 leader_election.cc:302] T 683d4041910f4355910e188c0b7df6c7 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:28.840610 31065 raft_consensus.cc:2781] T 683d4041910f4355910e188c0b7df6c7 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:28.840706 31065 raft_consensus.cc:482] T 683d4041910f4355910e188c0b7df6c7 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:28.840771 31065 raft_consensus.cc:3037] T 683d4041910f4355910e188c0b7df6c7 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:28.842741 31065 raft_consensus.cc:504] T 683d4041910f4355910e188c0b7df6c7 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:28.843240 31065 leader_election.cc:288] T 683d4041910f4355910e188c0b7df6c7 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Requested vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:12:28.843750 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "683d4041910f4355910e188c0b7df6c7" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:12:28.843775 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "683d4041910f4355910e188c0b7df6c7" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:12:28.843962 30005 raft_consensus.cc:3037] T 683d4041910f4355910e188c0b7df6c7 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:28.843983 30134 raft_consensus.cc:3037] T 683d4041910f4355910e188c0b7df6c7 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:28.846813 30134 raft_consensus.cc:2454] T 683d4041910f4355910e188c0b7df6c7 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:28.846832 30005 raft_consensus.cc:2454] T 683d4041910f4355910e188c0b7df6c7 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:28.847280 30199 leader_election.cc:302] T 683d4041910f4355910e188c0b7df6c7 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:28.847610 31065 raft_consensus.cc:2781] T 683d4041910f4355910e188c0b7df6c7 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:28.847750 31065 raft_consensus.cc:686] T 683d4041910f4355910e188c0b7df6c7 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Becoming Leader. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Running, Role: LEADER
I20240503 02:12:28.848018 31065 consensus_queue.cc:237] T 683d4041910f4355910e188c0b7df6c7 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:28.851778 29731 catalog_manager.cc:5484] T 683d4041910f4355910e188c0b7df6c7 P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4). New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:12:28.918470 31065 raft_consensus.cc:482] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:28.918691 31065 raft_consensus.cc:504] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:28.919265 31065 leader_election.cc:288] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:28.919778 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7527cc5931e24fa180efedbf81438efd" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:28.919798 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7527cc5931e24fa180efedbf81438efd" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:12:28.919982 30134 raft_consensus.cc:2454] T 7527cc5931e24fa180efedbf81438efd P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:28.920001 29875 raft_consensus.cc:2454] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:28.920367 30199 leader_election.cc:302] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:28.920667 31065 raft_consensus.cc:2781] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:28.920750 31065 raft_consensus.cc:482] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:28.920858 31065 raft_consensus.cc:3037] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:28.922739 31065 raft_consensus.cc:504] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:28.923352 31065 leader_election.cc:288] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Requested vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:28.923780 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7527cc5931e24fa180efedbf81438efd" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:12:28.923854 29874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7527cc5931e24fa180efedbf81438efd" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:12:28.923986 30134 raft_consensus.cc:3037] T 7527cc5931e24fa180efedbf81438efd P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:28.924058 29874 raft_consensus.cc:3037] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:28.925963 29874 raft_consensus.cc:2454] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:28.925981 30134 raft_consensus.cc:2454] T 7527cc5931e24fa180efedbf81438efd P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:28.926354 30199 leader_election.cc:302] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:28.926620 31065 raft_consensus.cc:2781] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:28.926781 31065 raft_consensus.cc:686] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Becoming Leader. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Running, Role: LEADER
I20240503 02:12:28.927054 31065 consensus_queue.cc:237] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:28.929826 29731 catalog_manager.cc:5484] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4). New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:12:29.253831 31120 consensus_queue.cc:1035] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:29.260126 31146 consensus_queue.cc:1035] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:29.294929 31144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:29.295250 31144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:29.302819 31144 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:12:29.310951 31065 consensus_queue.cc:1035] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:29.317137 31065 consensus_queue.cc:1035] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
I20240503 02:12:29.441412 30006 raft_consensus.cc:1261] T 683d4041910f4355910e188c0b7df6c7 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:29.442778 30136 raft_consensus.cc:1261] T 683d4041910f4355910e188c0b7df6c7 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:29.443440 31065 consensus_queue.cc:1035] T 683d4041910f4355910e188c0b7df6c7 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:29.447178 31065 consensus_queue.cc:1035] T 683d4041910f4355910e188c0b7df6c7 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:29.514552 31189 mvcc.cc:205] Tried to move back new op lower bound from 7023420823302836224 to 7023420820883824640. Current Snapshot: MvccSnapshot[applied={T|T < 7023420823302836224}]
I20240503 02:12:29.553906 31158 mvcc.cc:205] Tried to move back new op lower bound from 7023420823581339648 to 7023420820883824640. Current Snapshot: MvccSnapshot[applied={T|T < 7023420823581339648}]
I20240503 02:12:29.555873 31150 mvcc.cc:205] Tried to move back new op lower bound from 7023420823302836224 to 7023420820883824640. Current Snapshot: MvccSnapshot[applied={T|T < 7023420823302836224}]
INSERT report
rows total: 2000
time total: 208.862 ms
time per row: 0.104431 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 02:12:30.614622 29731 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:30.624322 29855 tablet_service.cc:1467] Processing CreateTablet for tablet 723a18a040ec41fa9d06459f4db808c7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e0ea42a974fb456485b832e1a5cb54ef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:30.624802 29855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 723a18a040ec41fa9d06459f4db808c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:30.624722 30114 tablet_service.cc:1467] Processing CreateTablet for tablet 723a18a040ec41fa9d06459f4db808c7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e0ea42a974fb456485b832e1a5cb54ef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:30.624797 29985 tablet_service.cc:1467] Processing CreateTablet for tablet 723a18a040ec41fa9d06459f4db808c7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e0ea42a974fb456485b832e1a5cb54ef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:30.625139 30114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 723a18a040ec41fa9d06459f4db808c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:30.625257 29985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 723a18a040ec41fa9d06459f4db808c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:30.626351 30246 tablet_service.cc:1467] Processing CreateTablet for tablet fa8e2df1f59e4b86a5c18067108e07bf (DEFAULT_TABLE table=rebalance_test_table_4 [id=e0ea42a974fb456485b832e1a5cb54ef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:30.626751 30246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa8e2df1f59e4b86a5c18067108e07bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:30.627862 30245 tablet_service.cc:1467] Processing CreateTablet for tablet 537c249dd4a44c23ade69a960b6e2675 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e0ea42a974fb456485b832e1a5cb54ef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:30.628223 30245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 537c249dd4a44c23ade69a960b6e2675. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:30.629961 30116 tablet_service.cc:1467] Processing CreateTablet for tablet fa8e2df1f59e4b86a5c18067108e07bf (DEFAULT_TABLE table=rebalance_test_table_4 [id=e0ea42a974fb456485b832e1a5cb54ef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:30.630391 30116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa8e2df1f59e4b86a5c18067108e07bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:30.634104 29853 tablet_service.cc:1467] Processing CreateTablet for tablet 537c249dd4a44c23ade69a960b6e2675 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e0ea42a974fb456485b832e1a5cb54ef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:30.634531 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 537c249dd4a44c23ade69a960b6e2675. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:30.637444 29854 tablet_service.cc:1467] Processing CreateTablet for tablet fa8e2df1f59e4b86a5c18067108e07bf (DEFAULT_TABLE table=rebalance_test_table_4 [id=e0ea42a974fb456485b832e1a5cb54ef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:30.637876 29854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa8e2df1f59e4b86a5c18067108e07bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:30.643708 29986 tablet_service.cc:1467] Processing CreateTablet for tablet 537c249dd4a44c23ade69a960b6e2675 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e0ea42a974fb456485b832e1a5cb54ef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:30.644382 29986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 537c249dd4a44c23ade69a960b6e2675. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:30.645972 31214 tablet_bootstrap.cc:492] T 723a18a040ec41fa9d06459f4db808c7 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:30.647747 31214 tablet_bootstrap.cc:654] T 723a18a040ec41fa9d06459f4db808c7 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:30.650315 31215 tablet_bootstrap.cc:492] T fa8e2df1f59e4b86a5c18067108e07bf P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:30.651712 31217 tablet_bootstrap.cc:492] T 537c249dd4a44c23ade69a960b6e2675 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:30.652887 31215 tablet_bootstrap.cc:654] T fa8e2df1f59e4b86a5c18067108e07bf P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:30.654381 31214 tablet_bootstrap.cc:492] T 723a18a040ec41fa9d06459f4db808c7 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:30.655720 31217 tablet_bootstrap.cc:654] T 537c249dd4a44c23ade69a960b6e2675 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:30.656028 31214 ts_tablet_manager.cc:1367] T 723a18a040ec41fa9d06459f4db808c7 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.001s
I20240503 02:12:30.657186 31214 raft_consensus.cc:348] T 723a18a040ec41fa9d06459f4db808c7 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:30.659032 31214 raft_consensus.cc:374] T 723a18a040ec41fa9d06459f4db808c7 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:30.659243 31214 raft_consensus.cc:729] T 723a18a040ec41fa9d06459f4db808c7 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:30.659685 31214 consensus_queue.cc:260] T 723a18a040ec41fa9d06459f4db808c7 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:30.661604 31214 ts_tablet_manager.cc:1398] T 723a18a040ec41fa9d06459f4db808c7 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 02:12:30.662014 31214 tablet_bootstrap.cc:492] T fa8e2df1f59e4b86a5c18067108e07bf P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:30.664242 31214 tablet_bootstrap.cc:654] T fa8e2df1f59e4b86a5c18067108e07bf P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:30.666455 31217 tablet_bootstrap.cc:492] T 537c249dd4a44c23ade69a960b6e2675 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:30.666747 31217 ts_tablet_manager.cc:1367] T 537c249dd4a44c23ade69a960b6e2675 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.005s
I20240503 02:12:30.668104 31217 raft_consensus.cc:348] T 537c249dd4a44c23ade69a960b6e2675 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:30.668707 31217 raft_consensus.cc:374] T 537c249dd4a44c23ade69a960b6e2675 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:30.668895 31217 raft_consensus.cc:729] T 537c249dd4a44c23ade69a960b6e2675 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:30.669265 31217 consensus_queue.cc:260] T 537c249dd4a44c23ade69a960b6e2675 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:30.670955 31217 ts_tablet_manager.cc:1398] T 537c249dd4a44c23ade69a960b6e2675 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:30.688652 31217 tablet_bootstrap.cc:492] T 723a18a040ec41fa9d06459f4db808c7 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:30.691206 31217 tablet_bootstrap.cc:654] T 723a18a040ec41fa9d06459f4db808c7 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:30.694324 31217 tablet_bootstrap.cc:492] T 723a18a040ec41fa9d06459f4db808c7 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:30.694684 31217 ts_tablet_manager.cc:1367] T 723a18a040ec41fa9d06459f4db808c7 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 02:12:30.695801 31217 raft_consensus.cc:348] T 723a18a040ec41fa9d06459f4db808c7 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:30.696393 31217 raft_consensus.cc:374] T 723a18a040ec41fa9d06459f4db808c7 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:30.696558 31217 raft_consensus.cc:729] T 723a18a040ec41fa9d06459f4db808c7 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:30.697001 31217 consensus_queue.cc:260] T 723a18a040ec41fa9d06459f4db808c7 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:30.698998 31217 ts_tablet_manager.cc:1398] T 723a18a040ec41fa9d06459f4db808c7 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:12:30.710279 31213 tablet_bootstrap.cc:492] T 723a18a040ec41fa9d06459f4db808c7 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:30.713219 31213 tablet_bootstrap.cc:654] T 723a18a040ec41fa9d06459f4db808c7 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:30.714267 31214 tablet_bootstrap.cc:492] T fa8e2df1f59e4b86a5c18067108e07bf P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:30.714548 31214 ts_tablet_manager.cc:1367] T fa8e2df1f59e4b86a5c18067108e07bf P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.053s user 0.004s sys 0.027s
I20240503 02:12:30.715327 31215 tablet_bootstrap.cc:492] T fa8e2df1f59e4b86a5c18067108e07bf P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:30.715495 31215 ts_tablet_manager.cc:1367] T fa8e2df1f59e4b86a5c18067108e07bf P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.065s user 0.019s sys 0.043s
I20240503 02:12:30.715348 31214 raft_consensus.cc:348] T fa8e2df1f59e4b86a5c18067108e07bf P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:30.715593 31214 raft_consensus.cc:374] T fa8e2df1f59e4b86a5c18067108e07bf P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:30.715678 31214 raft_consensus.cc:729] T fa8e2df1f59e4b86a5c18067108e07bf P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:30.715679 31213 tablet_bootstrap.cc:492] T 723a18a040ec41fa9d06459f4db808c7 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:30.715831 31213 ts_tablet_manager.cc:1367] T 723a18a040ec41fa9d06459f4db808c7 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 02:12:30.715970 31214 consensus_queue.cc:260] T fa8e2df1f59e4b86a5c18067108e07bf P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:30.716281 31215 raft_consensus.cc:348] T fa8e2df1f59e4b86a5c18067108e07bf P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:30.716486 31215 raft_consensus.cc:374] T fa8e2df1f59e4b86a5c18067108e07bf P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:30.716574 31215 raft_consensus.cc:729] T fa8e2df1f59e4b86a5c18067108e07bf P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:30.716565 31213 raft_consensus.cc:348] T 723a18a040ec41fa9d06459f4db808c7 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:30.716769 31213 raft_consensus.cc:374] T 723a18a040ec41fa9d06459f4db808c7 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:30.716859 31213 raft_consensus.cc:729] T 723a18a040ec41fa9d06459f4db808c7 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:30.716830 31215 consensus_queue.cc:260] T fa8e2df1f59e4b86a5c18067108e07bf P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:30.717147 31213 consensus_queue.cc:260] T 723a18a040ec41fa9d06459f4db808c7 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:30.717303 31214 ts_tablet_manager.cc:1398] T fa8e2df1f59e4b86a5c18067108e07bf P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:12:30.717425 31215 ts_tablet_manager.cc:1398] T fa8e2df1f59e4b86a5c18067108e07bf P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:30.717635 31213 ts_tablet_manager.cc:1398] T 723a18a040ec41fa9d06459f4db808c7 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:30.717715 31215 tablet_bootstrap.cc:492] T 537c249dd4a44c23ade69a960b6e2675 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:30.717896 31213 tablet_bootstrap.cc:492] T fa8e2df1f59e4b86a5c18067108e07bf P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:30.719345 31215 tablet_bootstrap.cc:654] T 537c249dd4a44c23ade69a960b6e2675 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:30.719553 31213 tablet_bootstrap.cc:654] T fa8e2df1f59e4b86a5c18067108e07bf P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:30.721275 31215 tablet_bootstrap.cc:492] T 537c249dd4a44c23ade69a960b6e2675 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:30.721448 31215 ts_tablet_manager.cc:1367] T 537c249dd4a44c23ade69a960b6e2675 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20240503 02:12:30.722139 31215 raft_consensus.cc:348] T 537c249dd4a44c23ade69a960b6e2675 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:30.722368 31215 raft_consensus.cc:374] T 537c249dd4a44c23ade69a960b6e2675 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:30.722446 31215 raft_consensus.cc:729] T 537c249dd4a44c23ade69a960b6e2675 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:30.722723 31213 tablet_bootstrap.cc:492] T fa8e2df1f59e4b86a5c18067108e07bf P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:30.722839 31213 ts_tablet_manager.cc:1367] T fa8e2df1f59e4b86a5c18067108e07bf P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:12:30.722693 31215 consensus_queue.cc:260] T 537c249dd4a44c23ade69a960b6e2675 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:30.723306 31215 ts_tablet_manager.cc:1398] T 537c249dd4a44c23ade69a960b6e2675 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:30.723583 31213 raft_consensus.cc:348] T fa8e2df1f59e4b86a5c18067108e07bf P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:30.723842 31213 raft_consensus.cc:374] T fa8e2df1f59e4b86a5c18067108e07bf P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:30.723945 31213 raft_consensus.cc:729] T fa8e2df1f59e4b86a5c18067108e07bf P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:30.724205 31213 consensus_queue.cc:260] T fa8e2df1f59e4b86a5c18067108e07bf P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:30.724874 31213 ts_tablet_manager.cc:1398] T fa8e2df1f59e4b86a5c18067108e07bf P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:30.725201 31213 tablet_bootstrap.cc:492] T 537c249dd4a44c23ade69a960b6e2675 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:30.726963 31213 tablet_bootstrap.cc:654] T 537c249dd4a44c23ade69a960b6e2675 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:30.729123 31213 tablet_bootstrap.cc:492] T 537c249dd4a44c23ade69a960b6e2675 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:30.729290 31213 ts_tablet_manager.cc:1367] T 537c249dd4a44c23ade69a960b6e2675 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 02:12:30.729933 31213 raft_consensus.cc:348] T 537c249dd4a44c23ade69a960b6e2675 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:30.730098 31213 raft_consensus.cc:374] T 537c249dd4a44c23ade69a960b6e2675 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:30.730175 31213 raft_consensus.cc:729] T 537c249dd4a44c23ade69a960b6e2675 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:30.730378 31213 consensus_queue.cc:260] T 537c249dd4a44c23ade69a960b6e2675 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:30.730806 31213 ts_tablet_manager.cc:1398] T 537c249dd4a44c23ade69a960b6e2675 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:12:30.762915 31146 raft_consensus.cc:482] T 723a18a040ec41fa9d06459f4db808c7 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:30.763144 31146 raft_consensus.cc:504] T 723a18a040ec41fa9d06459f4db808c7 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:30.763818 31146 leader_election.cc:288] T 723a18a040ec41fa9d06459f4db808c7 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:30.764427 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "723a18a040ec41fa9d06459f4db808c7" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:12:30.764493 29874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "723a18a040ec41fa9d06459f4db808c7" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:30.764724 29874 raft_consensus.cc:2454] T 723a18a040ec41fa9d06459f4db808c7 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:12:30.764710 30134 raft_consensus.cc:2454] T 723a18a040ec41fa9d06459f4db808c7 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:12:30.765192 29942 leader_election.cc:302] T 723a18a040ec41fa9d06459f4db808c7 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429; no voters:
I20240503 02:12:30.765522 31146 raft_consensus.cc:2781] T 723a18a040ec41fa9d06459f4db808c7 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:30.765635 31146 raft_consensus.cc:482] T 723a18a040ec41fa9d06459f4db808c7 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:30.765738 31146 raft_consensus.cc:3037] T 723a18a040ec41fa9d06459f4db808c7 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:30.767609 31146 raft_consensus.cc:504] T 723a18a040ec41fa9d06459f4db808c7 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:30.768286 31146 leader_election.cc:288] T 723a18a040ec41fa9d06459f4db808c7 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Requested vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:30.768812 29874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "723a18a040ec41fa9d06459f4db808c7" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:12:30.768857 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "723a18a040ec41fa9d06459f4db808c7" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:12:30.769083 30134 raft_consensus.cc:3037] T 723a18a040ec41fa9d06459f4db808c7 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:30.769062 29874 raft_consensus.cc:3037] T 723a18a040ec41fa9d06459f4db808c7 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:30.771920 30134 raft_consensus.cc:2454] T 723a18a040ec41fa9d06459f4db808c7 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:12:30.771939 29874 raft_consensus.cc:2454] T 723a18a040ec41fa9d06459f4db808c7 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:12:30.772392 29939 leader_election.cc:302] T 723a18a040ec41fa9d06459f4db808c7 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:30.772701 31146 raft_consensus.cc:2781] T 723a18a040ec41fa9d06459f4db808c7 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:30.773170 31146 raft_consensus.cc:686] T 723a18a040ec41fa9d06459f4db808c7 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Becoming Leader. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Running, Role: LEADER
I20240503 02:12:30.773500 31146 consensus_queue.cc:237] T 723a18a040ec41fa9d06459f4db808c7 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:30.776530 29731 catalog_manager.cc:5484] T 723a18a040ec41fa9d06459f4db808c7 P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: term changed from 0 to 1, leader changed from <none> to 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2). New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:12:30.812862 30941 raft_consensus.cc:482] T 537c249dd4a44c23ade69a960b6e2675 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:30.813093 30941 raft_consensus.cc:504] T 537c249dd4a44c23ade69a960b6e2675 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:30.813939 30941 leader_election.cc:288] T 537c249dd4a44c23ade69a960b6e2675 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:12:30.814520 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "537c249dd4a44c23ade69a960b6e2675" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:12:30.816149 30005 raft_consensus.cc:2454] T 537c249dd4a44c23ade69a960b6e2675 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:12:30.817183 29809 leader_election.cc:302] T 537c249dd4a44c23ade69a960b6e2675 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429; no voters:
I20240503 02:12:30.817742 30941 raft_consensus.cc:2781] T 537c249dd4a44c23ade69a960b6e2675 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:30.817853 30941 raft_consensus.cc:482] T 537c249dd4a44c23ade69a960b6e2675 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:30.817931 30941 raft_consensus.cc:3037] T 537c249dd4a44c23ade69a960b6e2675 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:30.821247 30941 raft_consensus.cc:504] T 537c249dd4a44c23ade69a960b6e2675 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:30.822922 30941 leader_election.cc:288] T 537c249dd4a44c23ade69a960b6e2675 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Requested vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:12:30.824335 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "537c249dd4a44c23ade69a960b6e2675" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:12:30.824682 30005 raft_consensus.cc:3037] T 537c249dd4a44c23ade69a960b6e2675 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:30.826406 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "537c249dd4a44c23ade69a960b6e2675" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:12:30.826517 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "537c249dd4a44c23ade69a960b6e2675" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:12:30.826666 30265 raft_consensus.cc:2454] T 537c249dd4a44c23ade69a960b6e2675 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:12:30.828052 30005 raft_consensus.cc:2454] T 537c249dd4a44c23ade69a960b6e2675 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:12:30.828637 29809 leader_election.cc:302] T 537c249dd4a44c23ade69a960b6e2675 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429; no voters: 1f10f80f13c84e80b03eda7085a3b012
I20240503 02:12:30.829020 30941 raft_consensus.cc:2781] T 537c249dd4a44c23ade69a960b6e2675 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:30.829483 30941 raft_consensus.cc:686] T 537c249dd4a44c23ade69a960b6e2675 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Becoming Leader. State: Replica: 16440e365430405fb10366ab70e2b429, State: Running, Role: LEADER
I20240503 02:12:30.830024 30941 consensus_queue.cc:237] T 537c249dd4a44c23ade69a960b6e2675 P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:30.837756 29731 catalog_manager.cc:5484] T 537c249dd4a44c23ade69a960b6e2675 P 16440e365430405fb10366ab70e2b429 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16440e365430405fb10366ab70e2b429 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:12:30.899334 30941 raft_consensus.cc:482] T fa8e2df1f59e4b86a5c18067108e07bf P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:30.899564 30941 raft_consensus.cc:504] T fa8e2df1f59e4b86a5c18067108e07bf P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:30.900214 30941 leader_election.cc:288] T fa8e2df1f59e4b86a5c18067108e07bf P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:30.900760 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8e2df1f59e4b86a5c18067108e07bf" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:12:30.900806 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8e2df1f59e4b86a5c18067108e07bf" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:12:30.900964 30134 raft_consensus.cc:2454] T fa8e2df1f59e4b86a5c18067108e07bf P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:12:30.901012 30265 raft_consensus.cc:2454] T fa8e2df1f59e4b86a5c18067108e07bf P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:12:30.901427 29808 leader_election.cc:302] T fa8e2df1f59e4b86a5c18067108e07bf P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:30.901763 30941 raft_consensus.cc:2781] T fa8e2df1f59e4b86a5c18067108e07bf P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:30.901876 30941 raft_consensus.cc:482] T fa8e2df1f59e4b86a5c18067108e07bf P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:30.901990 30941 raft_consensus.cc:3037] T fa8e2df1f59e4b86a5c18067108e07bf P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:30.903854 30941 raft_consensus.cc:504] T fa8e2df1f59e4b86a5c18067108e07bf P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:30.904428 30941 leader_election.cc:288] T fa8e2df1f59e4b86a5c18067108e07bf P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Requested vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:30.904839 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8e2df1f59e4b86a5c18067108e07bf" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:12:30.904896 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8e2df1f59e4b86a5c18067108e07bf" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:12:30.905050 30265 raft_consensus.cc:3037] T fa8e2df1f59e4b86a5c18067108e07bf P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:30.905077 30134 raft_consensus.cc:3037] T fa8e2df1f59e4b86a5c18067108e07bf P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:30.906883 30265 raft_consensus.cc:2454] T fa8e2df1f59e4b86a5c18067108e07bf P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:12:30.906883 30134 raft_consensus.cc:2454] T fa8e2df1f59e4b86a5c18067108e07bf P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:12:30.907279 29809 leader_election.cc:302] T fa8e2df1f59e4b86a5c18067108e07bf P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:30.907549 30941 raft_consensus.cc:2781] T fa8e2df1f59e4b86a5c18067108e07bf P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:30.907704 30941 raft_consensus.cc:686] T fa8e2df1f59e4b86a5c18067108e07bf P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Becoming Leader. State: Replica: 16440e365430405fb10366ab70e2b429, State: Running, Role: LEADER
I20240503 02:12:30.907999 30941 consensus_queue.cc:237] T fa8e2df1f59e4b86a5c18067108e07bf P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:30.910787 29731 catalog_manager.cc:5484] T fa8e2df1f59e4b86a5c18067108e07bf P 16440e365430405fb10366ab70e2b429 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16440e365430405fb10366ab70e2b429 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:12:31.157212 31146 consensus_queue.cc:1035] T 723a18a040ec41fa9d06459f4db808c7 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:31.163846 31146 consensus_queue.cc:1035] T 723a18a040ec41fa9d06459f4db808c7 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:31.309216 31234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:31.309574 31234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:31.317391 31234 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:12:31.339987 30265 raft_consensus.cc:3037] T 537c249dd4a44c23ade69a960b6e2675 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:31.343578 31231 consensus_queue.cc:1035] T 537c249dd4a44c23ade69a960b6e2675 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:31.348421 30941 consensus_queue.cc:1035] T 537c249dd4a44c23ade69a960b6e2675 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'
I20240503 02:12:31.454209 30941 consensus_queue.cc:1035] T fa8e2df1f59e4b86a5c18067108e07bf P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:31.497669 31277 consensus_queue.cc:1035] T fa8e2df1f59e4b86a5c18067108e07bf P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:31.578683 29809 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.2:37873, user_credentials={real_user=slave}} blocked reactor thread for 62458.5us
INSERT report
rows total: 2000
time total: 239.922 ms
time per row: 0.119961 ms
I20240503 02:12:31.727057 29731 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:31.736101 29985 tablet_service.cc:1467] Processing CreateTablet for tablet 8fdbaf1df10b46d5a47326415d0cd0e9 (DEFAULT_TABLE table=rebalance_test_table_5 [id=99dbec2a2b63467ab34b96ecc8479244]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:31.736423 30114 tablet_service.cc:1467] Processing CreateTablet for tablet 8fdbaf1df10b46d5a47326415d0cd0e9 (DEFAULT_TABLE table=rebalance_test_table_5 [id=99dbec2a2b63467ab34b96ecc8479244]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:31.736617 29985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fdbaf1df10b46d5a47326415d0cd0e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:31.736857 30114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fdbaf1df10b46d5a47326415d0cd0e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:31.737514 29855 tablet_service.cc:1467] Processing CreateTablet for tablet b7a2201f9605454ea12df4e3a7e5a63e (DEFAULT_TABLE table=rebalance_test_table_5 [id=99dbec2a2b63467ab34b96ecc8479244]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:31.737982 29855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7a2201f9605454ea12df4e3a7e5a63e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:31.738538 30245 tablet_service.cc:1467] Processing CreateTablet for tablet 8fdbaf1df10b46d5a47326415d0cd0e9 (DEFAULT_TABLE table=rebalance_test_table_5 [id=99dbec2a2b63467ab34b96ecc8479244]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:31.738946 30245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fdbaf1df10b46d5a47326415d0cd0e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:31.741822 29986 tablet_service.cc:1467] Processing CreateTablet for tablet 88b57085de3144cea3f64922ae5ccf55 (DEFAULT_TABLE table=rebalance_test_table_5 [id=99dbec2a2b63467ab34b96ecc8479244]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:31.742237 29986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88b57085de3144cea3f64922ae5ccf55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:31.744174 30246 tablet_service.cc:1467] Processing CreateTablet for tablet 88b57085de3144cea3f64922ae5ccf55 (DEFAULT_TABLE table=rebalance_test_table_5 [id=99dbec2a2b63467ab34b96ecc8479244]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:31.744585 30246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88b57085de3144cea3f64922ae5ccf55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:31.745688 29984 tablet_service.cc:1467] Processing CreateTablet for tablet b7a2201f9605454ea12df4e3a7e5a63e (DEFAULT_TABLE table=rebalance_test_table_5 [id=99dbec2a2b63467ab34b96ecc8479244]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:31.746045 29984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7a2201f9605454ea12df4e3a7e5a63e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:31.750185 30116 tablet_service.cc:1467] Processing CreateTablet for tablet 88b57085de3144cea3f64922ae5ccf55 (DEFAULT_TABLE table=rebalance_test_table_5 [id=99dbec2a2b63467ab34b96ecc8479244]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:31.750620 30116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88b57085de3144cea3f64922ae5ccf55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:31.751638 31298 tablet_bootstrap.cc:492] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:31.752921 31298 tablet_bootstrap.cc:654] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:31.753423 30115 tablet_service.cc:1467] Processing CreateTablet for tablet b7a2201f9605454ea12df4e3a7e5a63e (DEFAULT_TABLE table=rebalance_test_table_5 [id=99dbec2a2b63467ab34b96ecc8479244]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:31.753785 30115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7a2201f9605454ea12df4e3a7e5a63e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:31.760851 31300 tablet_bootstrap.cc:492] T b7a2201f9605454ea12df4e3a7e5a63e P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:31.762847 31300 tablet_bootstrap.cc:654] T b7a2201f9605454ea12df4e3a7e5a63e P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:31.768136 31302 tablet_bootstrap.cc:492] T 88b57085de3144cea3f64922ae5ccf55 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:31.768920 31297 tablet_bootstrap.cc:492] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:31.769147 31298 tablet_bootstrap.cc:492] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:31.769889 31298 ts_tablet_manager.cc:1367] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.008s
I20240503 02:12:31.770591 31302 tablet_bootstrap.cc:654] T 88b57085de3144cea3f64922ae5ccf55 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:31.770887 31297 tablet_bootstrap.cc:654] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:31.771035 31298 raft_consensus.cc:348] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:31.771354 31298 raft_consensus.cc:374] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:31.771497 31298 raft_consensus.cc:729] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:31.771680 31300 tablet_bootstrap.cc:492] T b7a2201f9605454ea12df4e3a7e5a63e P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:31.771840 31300 ts_tablet_manager.cc:1367] T b7a2201f9605454ea12df4e3a7e5a63e P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240503 02:12:31.772028 31298 consensus_queue.cc:260] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:31.772748 31300 raft_consensus.cc:348] T b7a2201f9605454ea12df4e3a7e5a63e P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:31.773003 31300 raft_consensus.cc:374] T b7a2201f9605454ea12df4e3a7e5a63e P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:31.773504 31300 raft_consensus.cc:729] T b7a2201f9605454ea12df4e3a7e5a63e P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:31.773885 31300 consensus_queue.cc:260] T b7a2201f9605454ea12df4e3a7e5a63e P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:31.775609 31300 ts_tablet_manager.cc:1398] T b7a2201f9605454ea12df4e3a7e5a63e P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:12:31.776386 31298 ts_tablet_manager.cc:1398] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 02:12:31.776722 31298 tablet_bootstrap.cc:492] T 88b57085de3144cea3f64922ae5ccf55 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:31.777061 31297 tablet_bootstrap.cc:492] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:31.777921 31297 ts_tablet_manager.cc:1367] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20240503 02:12:31.778394 31298 tablet_bootstrap.cc:654] T 88b57085de3144cea3f64922ae5ccf55 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:31.778765 31297 raft_consensus.cc:348] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:31.778980 31297 raft_consensus.cc:374] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:31.779081 31297 raft_consensus.cc:729] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:31.779320 31297 consensus_queue.cc:260] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:31.779942 31297 ts_tablet_manager.cc:1398] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:31.780606 31297 tablet_bootstrap.cc:492] T b7a2201f9605454ea12df4e3a7e5a63e P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:31.782258 31297 tablet_bootstrap.cc:654] T b7a2201f9605454ea12df4e3a7e5a63e P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:31.786661 31281 raft_consensus.cc:482] T b7a2201f9605454ea12df4e3a7e5a63e P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:31.786850 31281 raft_consensus.cc:504] T b7a2201f9605454ea12df4e3a7e5a63e P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:31.787801 31281 leader_election.cc:288] T b7a2201f9605454ea12df4e3a7e5a63e P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:31.788008 31302 tablet_bootstrap.cc:492] T 88b57085de3144cea3f64922ae5ccf55 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:31.788215 31302 ts_tablet_manager.cc:1367] T 88b57085de3144cea3f64922ae5ccf55 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.005s
I20240503 02:12:31.788280 31298 tablet_bootstrap.cc:492] T 88b57085de3144cea3f64922ae5ccf55 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:31.788281 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7a2201f9605454ea12df4e3a7e5a63e" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:12:31.788523 31298 ts_tablet_manager.cc:1367] T 88b57085de3144cea3f64922ae5ccf55 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20240503 02:12:31.788564 30135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7a2201f9605454ea12df4e3a7e5a63e" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
W20240503 02:12:31.788995 29809 leader_election.cc:341] T b7a2201f9605454ea12df4e3a7e5a63e P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:31.789394 31298 raft_consensus.cc:348] T 88b57085de3144cea3f64922ae5ccf55 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:31.789628 31298 raft_consensus.cc:374] T 88b57085de3144cea3f64922ae5ccf55 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:31.789710 31298 raft_consensus.cc:729] T 88b57085de3144cea3f64922ae5ccf55 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:31.789567 31302 raft_consensus.cc:348] T 88b57085de3144cea3f64922ae5ccf55 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:31.789963 31298 consensus_queue.cc:260] T 88b57085de3144cea3f64922ae5ccf55 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:31.790091 31302 raft_consensus.cc:374] T 88b57085de3144cea3f64922ae5ccf55 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:31.790287 31302 raft_consensus.cc:729] T 88b57085de3144cea3f64922ae5ccf55 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:31.790592 31298 ts_tablet_manager.cc:1398] T 88b57085de3144cea3f64922ae5ccf55 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:31.790740 31302 consensus_queue.cc:260] T 88b57085de3144cea3f64922ae5ccf55 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
W20240503 02:12:31.791039 29808 leader_election.cc:341] T b7a2201f9605454ea12df4e3a7e5a63e P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:31.791190 29808 leader_election.cc:302] T b7a2201f9605454ea12df4e3a7e5a63e P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 16440e365430405fb10366ab70e2b429; no voters: 155735c7f8b14997ac4c770aee2a9401, a9094240860e43bfb129193f3bfd5be6
I20240503 02:12:31.791352 31298 tablet_bootstrap.cc:492] T b7a2201f9605454ea12df4e3a7e5a63e P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:31.791544 31302 ts_tablet_manager.cc:1398] T 88b57085de3144cea3f64922ae5ccf55 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:31.791910 31302 tablet_bootstrap.cc:492] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:31.792924 31281 raft_consensus.cc:2726] T b7a2201f9605454ea12df4e3a7e5a63e P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:31.792969 31298 tablet_bootstrap.cc:654] T b7a2201f9605454ea12df4e3a7e5a63e P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:31.793767 31302 tablet_bootstrap.cc:654] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:31.795643 31297 tablet_bootstrap.cc:492] T b7a2201f9605454ea12df4e3a7e5a63e P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:31.795828 31297 ts_tablet_manager.cc:1367] T b7a2201f9605454ea12df4e3a7e5a63e P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
I20240503 02:12:31.796651 31297 raft_consensus.cc:348] T b7a2201f9605454ea12df4e3a7e5a63e P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:31.796896 31297 raft_consensus.cc:374] T b7a2201f9605454ea12df4e3a7e5a63e P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:31.796967 31297 raft_consensus.cc:729] T b7a2201f9605454ea12df4e3a7e5a63e P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:31.797097 31298 tablet_bootstrap.cc:492] T b7a2201f9605454ea12df4e3a7e5a63e P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:31.797242 31298 ts_tablet_manager.cc:1367] T b7a2201f9605454ea12df4e3a7e5a63e P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 02:12:31.798008 31298 raft_consensus.cc:348] T b7a2201f9605454ea12df4e3a7e5a63e P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:31.798199 31298 raft_consensus.cc:374] T b7a2201f9605454ea12df4e3a7e5a63e P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:31.798290 31298 raft_consensus.cc:729] T b7a2201f9605454ea12df4e3a7e5a63e P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:31.798511 31298 consensus_queue.cc:260] T b7a2201f9605454ea12df4e3a7e5a63e P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:31.799042 31298 ts_tablet_manager.cc:1398] T b7a2201f9605454ea12df4e3a7e5a63e P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:31.800493 30869 raft_consensus.cc:482] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:31.801136 31302 tablet_bootstrap.cc:492] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:31.801059 30869 raft_consensus.cc:504] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:31.801314 31302 ts_tablet_manager.cc:1367] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.001s
I20240503 02:12:31.801671 30869 leader_election.cc:288] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:12:31.802173 31302 raft_consensus.cc:348] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:31.802405 31302 raft_consensus.cc:374] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:31.797245 31297 consensus_queue.cc:260] T b7a2201f9605454ea12df4e3a7e5a63e P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:31.802496 31302 raft_consensus.cc:729] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:31.802762 31302 consensus_queue.cc:260] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:31.803004 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fdbaf1df10b46d5a47326415d0cd0e9" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:12:31.803234 30005 raft_consensus.cc:2454] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 0.
I20240503 02:12:31.803287 31297 ts_tablet_manager.cc:1398] T b7a2201f9605454ea12df4e3a7e5a63e P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:12:31.803341 31302 ts_tablet_manager.cc:1398] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:31.803612 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fdbaf1df10b46d5a47326415d0cd0e9" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:12:31.803761 31297 tablet_bootstrap.cc:492] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:31.803853 30265 raft_consensus.cc:2454] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 0.
I20240503 02:12:31.804314 30070 leader_election.cc:302] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:31.804610 30869 raft_consensus.cc:2781] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:31.804731 30869 raft_consensus.cc:482] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:31.804860 30869 raft_consensus.cc:3037] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:31.805950 31297 tablet_bootstrap.cc:654] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:31.807875 30869 raft_consensus.cc:504] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:31.809247 31297 tablet_bootstrap.cc:492] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:31.809518 30869 leader_election.cc:288] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Requested vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:12:31.809823 31297 ts_tablet_manager.cc:1367] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20240503 02:12:31.809984 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fdbaf1df10b46d5a47326415d0cd0e9" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:12:31.810227 30005 raft_consensus.cc:3037] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:31.810907 31297 raft_consensus.cc:348] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:31.811161 31297 raft_consensus.cc:374] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:31.811283 31297 raft_consensus.cc:729] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:31.811558 31297 consensus_queue.cc:260] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:31.812917 31297 ts_tablet_manager.cc:1398] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:31.812988 30005 raft_consensus.cc:2454] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 1.
I20240503 02:12:31.813431 30070 leader_election.cc:302] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:31.813501 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fdbaf1df10b46d5a47326415d0cd0e9" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:12:31.813712 30265 raft_consensus.cc:3037] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:31.813794 30869 raft_consensus.cc:2781] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:31.817250 30265 raft_consensus.cc:2454] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 1.
I20240503 02:12:31.818372 30869 raft_consensus.cc:686] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Becoming Leader. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Running, Role: LEADER
I20240503 02:12:31.818738 30869 consensus_queue.cc:237] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:31.821835 30869 raft_consensus.cc:482] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:31.821967 30869 raft_consensus.cc:504] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:31.821779 29726 catalog_manager.cc:5484] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9094240860e43bfb129193f3bfd5be6 (127.22.108.3). New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:12:31.825387 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88b57085de3144cea3f64922ae5ccf55" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:12:31.825613 30005 raft_consensus.cc:2454] T 88b57085de3144cea3f64922ae5ccf55 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 0.
I20240503 02:12:31.826081 30070 leader_election.cc:302] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:31.826359 31310 raft_consensus.cc:2781] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:31.826467 31310 raft_consensus.cc:482] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:31.826591 31310 raft_consensus.cc:3037] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:31.826680 30869 leader_election.cc:288] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:12:31.827306 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88b57085de3144cea3f64922ae5ccf55" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:12:31.827567 30265 raft_consensus.cc:2454] T 88b57085de3144cea3f64922ae5ccf55 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 0.
I20240503 02:12:31.846974 31310 raft_consensus.cc:504] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:31.847595 31310 leader_election.cc:288] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Requested vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:12:31.848171 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88b57085de3144cea3f64922ae5ccf55" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:12:31.848217 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88b57085de3144cea3f64922ae5ccf55" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:12:31.848356 30005 raft_consensus.cc:3037] T 88b57085de3144cea3f64922ae5ccf55 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:31.848469 30265 raft_consensus.cc:3037] T 88b57085de3144cea3f64922ae5ccf55 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:31.850425 30005 raft_consensus.cc:2454] T 88b57085de3144cea3f64922ae5ccf55 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 1.
I20240503 02:12:31.850484 30265 raft_consensus.cc:2454] T 88b57085de3144cea3f64922ae5ccf55 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 1.
I20240503 02:12:31.850921 30070 leader_election.cc:302] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:31.851229 31310 raft_consensus.cc:2781] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:31.851400 31310 raft_consensus.cc:686] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Becoming Leader. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Running, Role: LEADER
I20240503 02:12:31.851672 31310 consensus_queue.cc:237] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:31.855036 29726 catalog_manager.cc:5484] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9094240860e43bfb129193f3bfd5be6 (127.22.108.3). New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:12:32.210367 31310 consensus_queue.cc:1035] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:32.216163 31310 consensus_queue.cc:1035] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:32.217588 31227 raft_consensus.cc:482] T b7a2201f9605454ea12df4e3a7e5a63e P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:32.217777 31227 raft_consensus.cc:504] T b7a2201f9605454ea12df4e3a7e5a63e P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:32.218525 31227 leader_election.cc:288] T b7a2201f9605454ea12df4e3a7e5a63e P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:32.219035 29874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7a2201f9605454ea12df4e3a7e5a63e" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:32.219241 29874 raft_consensus.cc:2454] T b7a2201f9605454ea12df4e3a7e5a63e P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:12:32.219228 30135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7a2201f9605454ea12df4e3a7e5a63e" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:12:32.219422 30135 raft_consensus.cc:2454] T b7a2201f9605454ea12df4e3a7e5a63e P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:12:32.219803 29939 leader_election.cc:302] T b7a2201f9605454ea12df4e3a7e5a63e P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:32.220075 31227 raft_consensus.cc:2781] T b7a2201f9605454ea12df4e3a7e5a63e P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:32.220191 31227 raft_consensus.cc:482] T b7a2201f9605454ea12df4e3a7e5a63e P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:32.220433 31227 raft_consensus.cc:3037] T b7a2201f9605454ea12df4e3a7e5a63e P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:32.246940 31227 raft_consensus.cc:504] T b7a2201f9605454ea12df4e3a7e5a63e P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:32.247589 31227 leader_election.cc:288] T b7a2201f9605454ea12df4e3a7e5a63e P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Requested vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:32.248142 29874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7a2201f9605454ea12df4e3a7e5a63e" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:12:32.248163 30135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7a2201f9605454ea12df4e3a7e5a63e" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:12:32.248351 30135 raft_consensus.cc:3037] T b7a2201f9605454ea12df4e3a7e5a63e P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:32.248354 29874 raft_consensus.cc:3037] T b7a2201f9605454ea12df4e3a7e5a63e P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:32.250249 30135 raft_consensus.cc:2454] T b7a2201f9605454ea12df4e3a7e5a63e P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:12:32.250304 29874 raft_consensus.cc:2454] T b7a2201f9605454ea12df4e3a7e5a63e P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:12:32.250708 29939 leader_election.cc:302] T b7a2201f9605454ea12df4e3a7e5a63e P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:32.251035 31227 raft_consensus.cc:2781] T b7a2201f9605454ea12df4e3a7e5a63e P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:32.251360 31227 raft_consensus.cc:686] T b7a2201f9605454ea12df4e3a7e5a63e P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Becoming Leader. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Running, Role: LEADER
I20240503 02:12:32.251618 31227 consensus_queue.cc:237] T b7a2201f9605454ea12df4e3a7e5a63e P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:32.254242 29731 catalog_manager.cc:5484] T b7a2201f9605454ea12df4e3a7e5a63e P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: term changed from 0 to 1, leader changed from <none> to 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2). New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:12:32.360291 31310 consensus_queue.cc:1035] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:32.366838 30869 consensus_queue.cc:1035] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:32.670143 31321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:32.670455 31321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:32.677992 31321 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:12:32.680100 31227 consensus_queue.cc:1035] T b7a2201f9605454ea12df4e3a7e5a63e P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:32.689190 31319 consensus_queue.cc:1035] T b7a2201f9605454ea12df4e3a7e5a63e P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_5'
INSERT report
rows total: 2000
time total: 253.13 ms
time per row: 0.126565 ms
I20240503 02:12:33.084101 29732 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:33.093626 29855 tablet_service.cc:1467] Processing CreateTablet for tablet 636257ce6f0f43cdb764aae55a9e764e (DEFAULT_TABLE table=rebalance_test_table_6 [id=fdf8a5db2b9b450b8db9d7a71433d20b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.093776 29984 tablet_service.cc:1467] Processing CreateTablet for tablet 636257ce6f0f43cdb764aae55a9e764e (DEFAULT_TABLE table=rebalance_test_table_6 [id=fdf8a5db2b9b450b8db9d7a71433d20b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.094162 29855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 636257ce6f0f43cdb764aae55a9e764e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.094221 29984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 636257ce6f0f43cdb764aae55a9e764e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.094887 29854 tablet_service.cc:1467] Processing CreateTablet for tablet d4a5af9181d0429a8a4f4d734c59e2b6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fdf8a5db2b9b450b8db9d7a71433d20b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.095297 29854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4a5af9181d0429a8a4f4d734c59e2b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.095626 30244 tablet_service.cc:1467] Processing CreateTablet for tablet 8a9e3025d1764ce1aa48071ac3e5d902 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fdf8a5db2b9b450b8db9d7a71433d20b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.096074 30244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a9e3025d1764ce1aa48071ac3e5d902. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.096185 30246 tablet_service.cc:1467] Processing CreateTablet for tablet 636257ce6f0f43cdb764aae55a9e764e (DEFAULT_TABLE table=rebalance_test_table_6 [id=fdf8a5db2b9b450b8db9d7a71433d20b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.096522 30246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 636257ce6f0f43cdb764aae55a9e764e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.096503 29853 tablet_service.cc:1467] Processing CreateTablet for tablet 8a9e3025d1764ce1aa48071ac3e5d902 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fdf8a5db2b9b450b8db9d7a71433d20b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.096853 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a9e3025d1764ce1aa48071ac3e5d902. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.097702 29986 tablet_service.cc:1467] Processing CreateTablet for tablet 8a9e3025d1764ce1aa48071ac3e5d902 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fdf8a5db2b9b450b8db9d7a71433d20b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.098078 29986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a9e3025d1764ce1aa48071ac3e5d902. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.101841 30245 tablet_service.cc:1467] Processing CreateTablet for tablet d4a5af9181d0429a8a4f4d734c59e2b6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fdf8a5db2b9b450b8db9d7a71433d20b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.102255 30245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4a5af9181d0429a8a4f4d734c59e2b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.105679 30114 tablet_service.cc:1467] Processing CreateTablet for tablet d4a5af9181d0429a8a4f4d734c59e2b6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fdf8a5db2b9b450b8db9d7a71433d20b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.106253 30114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4a5af9181d0429a8a4f4d734c59e2b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.115895 31375 tablet_bootstrap.cc:492] T 8a9e3025d1764ce1aa48071ac3e5d902 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:33.116724 31377 tablet_bootstrap.cc:492] T 8a9e3025d1764ce1aa48071ac3e5d902 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:33.118471 31375 tablet_bootstrap.cc:654] T 8a9e3025d1764ce1aa48071ac3e5d902 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.119967 31377 tablet_bootstrap.cc:654] T 8a9e3025d1764ce1aa48071ac3e5d902 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.130507 31377 tablet_bootstrap.cc:492] T 8a9e3025d1764ce1aa48071ac3e5d902 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:33.130782 31377 ts_tablet_manager.cc:1367] T 8a9e3025d1764ce1aa48071ac3e5d902 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20240503 02:12:33.131680 31377 raft_consensus.cc:348] T 8a9e3025d1764ce1aa48071ac3e5d902 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:33.131997 31377 raft_consensus.cc:374] T 8a9e3025d1764ce1aa48071ac3e5d902 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.132134 31377 raft_consensus.cc:729] T 8a9e3025d1764ce1aa48071ac3e5d902 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.132403 31377 consensus_queue.cc:260] T 8a9e3025d1764ce1aa48071ac3e5d902 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:33.133164 31377 ts_tablet_manager.cc:1398] T 8a9e3025d1764ce1aa48071ac3e5d902 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:33.133524 31377 tablet_bootstrap.cc:492] T 636257ce6f0f43cdb764aae55a9e764e P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:33.135531 31377 tablet_bootstrap.cc:654] T 636257ce6f0f43cdb764aae55a9e764e P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.135980 31380 tablet_bootstrap.cc:492] T d4a5af9181d0429a8a4f4d734c59e2b6 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:33.138332 31380 tablet_bootstrap.cc:654] T d4a5af9181d0429a8a4f4d734c59e2b6 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.142833 31376 tablet_bootstrap.cc:492] T 8a9e3025d1764ce1aa48071ac3e5d902 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:33.142957 31377 tablet_bootstrap.cc:492] T 636257ce6f0f43cdb764aae55a9e764e P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:33.143488 31377 ts_tablet_manager.cc:1367] T 636257ce6f0f43cdb764aae55a9e764e P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.001s
I20240503 02:12:33.144294 31375 tablet_bootstrap.cc:492] T 8a9e3025d1764ce1aa48071ac3e5d902 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:33.144444 31375 ts_tablet_manager.cc:1367] T 8a9e3025d1764ce1aa48071ac3e5d902 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.029s user 0.014s sys 0.007s
I20240503 02:12:33.144372 31377 raft_consensus.cc:348] T 636257ce6f0f43cdb764aae55a9e764e P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:33.144613 31377 raft_consensus.cc:374] T 636257ce6f0f43cdb764aae55a9e764e P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.144706 31377 raft_consensus.cc:729] T 636257ce6f0f43cdb764aae55a9e764e P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.145040 31377 consensus_queue.cc:260] T 636257ce6f0f43cdb764aae55a9e764e P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:33.145607 31377 ts_tablet_manager.cc:1398] T 636257ce6f0f43cdb764aae55a9e764e P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:12:33.146027 31376 tablet_bootstrap.cc:654] T 8a9e3025d1764ce1aa48071ac3e5d902 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.145311 31375 raft_consensus.cc:348] T 8a9e3025d1764ce1aa48071ac3e5d902 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:33.146194 31375 raft_consensus.cc:374] T 8a9e3025d1764ce1aa48071ac3e5d902 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.146327 31375 raft_consensus.cc:729] T 8a9e3025d1764ce1aa48071ac3e5d902 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.146601 31375 consensus_queue.cc:260] T 8a9e3025d1764ce1aa48071ac3e5d902 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:33.147145 31375 ts_tablet_manager.cc:1398] T 8a9e3025d1764ce1aa48071ac3e5d902 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:33.147503 31375 tablet_bootstrap.cc:492] T d4a5af9181d0429a8a4f4d734c59e2b6 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:33.149048 31380 tablet_bootstrap.cc:492] T d4a5af9181d0429a8a4f4d734c59e2b6 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:33.149221 31380 ts_tablet_manager.cc:1367] T d4a5af9181d0429a8a4f4d734c59e2b6 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20240503 02:12:33.149303 31375 tablet_bootstrap.cc:654] T d4a5af9181d0429a8a4f4d734c59e2b6 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.150071 31380 raft_consensus.cc:348] T d4a5af9181d0429a8a4f4d734c59e2b6 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:33.150363 31380 raft_consensus.cc:374] T d4a5af9181d0429a8a4f4d734c59e2b6 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.150496 31380 raft_consensus.cc:729] T d4a5af9181d0429a8a4f4d734c59e2b6 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.150938 31380 consensus_queue.cc:260] T d4a5af9181d0429a8a4f4d734c59e2b6 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:33.151508 31376 tablet_bootstrap.cc:492] T 8a9e3025d1764ce1aa48071ac3e5d902 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:33.151558 31375 tablet_bootstrap.cc:492] T d4a5af9181d0429a8a4f4d734c59e2b6 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:33.151665 31376 ts_tablet_manager.cc:1367] T 8a9e3025d1764ce1aa48071ac3e5d902 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240503 02:12:33.151705 31375 ts_tablet_manager.cc:1367] T d4a5af9181d0429a8a4f4d734c59e2b6 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:12:33.152390 31375 raft_consensus.cc:348] T d4a5af9181d0429a8a4f4d734c59e2b6 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:33.152475 31376 raft_consensus.cc:348] T 8a9e3025d1764ce1aa48071ac3e5d902 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:33.152655 31375 raft_consensus.cc:374] T d4a5af9181d0429a8a4f4d734c59e2b6 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.152736 31380 ts_tablet_manager.cc:1398] T d4a5af9181d0429a8a4f4d734c59e2b6 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:33.152737 31376 raft_consensus.cc:374] T 8a9e3025d1764ce1aa48071ac3e5d902 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.152762 31375 raft_consensus.cc:729] T d4a5af9181d0429a8a4f4d734c59e2b6 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.152835 31376 raft_consensus.cc:729] T 8a9e3025d1764ce1aa48071ac3e5d902 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.153131 31375 consensus_queue.cc:260] T d4a5af9181d0429a8a4f4d734c59e2b6 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:33.153406 31376 consensus_queue.cc:260] T 8a9e3025d1764ce1aa48071ac3e5d902 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:33.154048 31376 ts_tablet_manager.cc:1398] T 8a9e3025d1764ce1aa48071ac3e5d902 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:33.154352 31376 tablet_bootstrap.cc:492] T 636257ce6f0f43cdb764aae55a9e764e P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:33.154531 31375 ts_tablet_manager.cc:1398] T d4a5af9181d0429a8a4f4d734c59e2b6 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:33.154819 31375 tablet_bootstrap.cc:492] T 636257ce6f0f43cdb764aae55a9e764e P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:33.156004 31376 tablet_bootstrap.cc:654] T 636257ce6f0f43cdb764aae55a9e764e P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.157109 31375 tablet_bootstrap.cc:654] T 636257ce6f0f43cdb764aae55a9e764e P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.163448 31375 tablet_bootstrap.cc:492] T 636257ce6f0f43cdb764aae55a9e764e P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:33.163560 31376 tablet_bootstrap.cc:492] T 636257ce6f0f43cdb764aae55a9e764e P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:33.163580 31375 ts_tablet_manager.cc:1367] T 636257ce6f0f43cdb764aae55a9e764e P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240503 02:12:33.163754 31376 ts_tablet_manager.cc:1367] T 636257ce6f0f43cdb764aae55a9e764e P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.007s
I20240503 02:12:33.164254 31375 raft_consensus.cc:348] T 636257ce6f0f43cdb764aae55a9e764e P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:33.164407 31375 raft_consensus.cc:374] T 636257ce6f0f43cdb764aae55a9e764e P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.164476 31375 raft_consensus.cc:729] T 636257ce6f0f43cdb764aae55a9e764e P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.164366 31376 raft_consensus.cc:348] T 636257ce6f0f43cdb764aae55a9e764e P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:33.164780 31376 raft_consensus.cc:374] T 636257ce6f0f43cdb764aae55a9e764e P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.164712 31375 consensus_queue.cc:260] T 636257ce6f0f43cdb764aae55a9e764e P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:33.164873 31376 raft_consensus.cc:729] T 636257ce6f0f43cdb764aae55a9e764e P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.165144 31376 consensus_queue.cc:260] T 636257ce6f0f43cdb764aae55a9e764e P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:33.165385 31375 ts_tablet_manager.cc:1398] T 636257ce6f0f43cdb764aae55a9e764e P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:33.165736 31376 ts_tablet_manager.cc:1398] T 636257ce6f0f43cdb764aae55a9e764e P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:33.166023 31376 tablet_bootstrap.cc:492] T d4a5af9181d0429a8a4f4d734c59e2b6 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:33.167312 31376 tablet_bootstrap.cc:654] T d4a5af9181d0429a8a4f4d734c59e2b6 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.169674 31376 tablet_bootstrap.cc:492] T d4a5af9181d0429a8a4f4d734c59e2b6 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:33.169831 31376 ts_tablet_manager.cc:1367] T d4a5af9181d0429a8a4f4d734c59e2b6 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240503 02:12:33.170589 31376 raft_consensus.cc:348] T d4a5af9181d0429a8a4f4d734c59e2b6 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:33.170758 31376 raft_consensus.cc:374] T d4a5af9181d0429a8a4f4d734c59e2b6 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.170862 31376 raft_consensus.cc:729] T d4a5af9181d0429a8a4f4d734c59e2b6 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.171069 31376 consensus_queue.cc:260] T d4a5af9181d0429a8a4f4d734c59e2b6 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:33.171525 31376 ts_tablet_manager.cc:1398] T d4a5af9181d0429a8a4f4d734c59e2b6 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:33.208596 31221 raft_consensus.cc:482] T 8a9e3025d1764ce1aa48071ac3e5d902 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:33.208774 31221 raft_consensus.cc:504] T 8a9e3025d1764ce1aa48071ac3e5d902 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:33.209491 31221 leader_election.cc:288] T 8a9e3025d1764ce1aa48071ac3e5d902 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:33.209908 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9e3025d1764ce1aa48071ac3e5d902" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:12:33.210017 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9e3025d1764ce1aa48071ac3e5d902" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:33.210170 30005 raft_consensus.cc:2454] T 8a9e3025d1764ce1aa48071ac3e5d902 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:33.210260 29875 raft_consensus.cc:2454] T 8a9e3025d1764ce1aa48071ac3e5d902 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:33.210606 30200 leader_election.cc:302] T 8a9e3025d1764ce1aa48071ac3e5d902 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:33.210892 31221 raft_consensus.cc:2781] T 8a9e3025d1764ce1aa48071ac3e5d902 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:33.211033 31221 raft_consensus.cc:482] T 8a9e3025d1764ce1aa48071ac3e5d902 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:33.211129 31221 raft_consensus.cc:3037] T 8a9e3025d1764ce1aa48071ac3e5d902 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.214116 31221 raft_consensus.cc:504] T 8a9e3025d1764ce1aa48071ac3e5d902 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:33.214684 31221 leader_election.cc:288] T 8a9e3025d1764ce1aa48071ac3e5d902 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Requested vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:33.215108 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9e3025d1764ce1aa48071ac3e5d902" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:12:33.215216 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9e3025d1764ce1aa48071ac3e5d902" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:12:33.215360 30005 raft_consensus.cc:3037] T 8a9e3025d1764ce1aa48071ac3e5d902 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.215433 29875 raft_consensus.cc:3037] T 8a9e3025d1764ce1aa48071ac3e5d902 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.218216 30005 raft_consensus.cc:2454] T 8a9e3025d1764ce1aa48071ac3e5d902 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:33.218216 29875 raft_consensus.cc:2454] T 8a9e3025d1764ce1aa48071ac3e5d902 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:33.218642 30200 leader_election.cc:302] T 8a9e3025d1764ce1aa48071ac3e5d902 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:33.218983 31221 raft_consensus.cc:2781] T 8a9e3025d1764ce1aa48071ac3e5d902 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:33.219372 31221 raft_consensus.cc:686] T 8a9e3025d1764ce1aa48071ac3e5d902 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Becoming Leader. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Running, Role: LEADER
I20240503 02:12:33.219648 31221 consensus_queue.cc:237] T 8a9e3025d1764ce1aa48071ac3e5d902 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:33.222425 29731 catalog_manager.cc:5484] T 8a9e3025d1764ce1aa48071ac3e5d902 P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4). New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:12:33.349481 31221 raft_consensus.cc:482] T 636257ce6f0f43cdb764aae55a9e764e P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:33.349716 31221 raft_consensus.cc:504] T 636257ce6f0f43cdb764aae55a9e764e P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:33.350461 31221 leader_election.cc:288] T 636257ce6f0f43cdb764aae55a9e764e P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:12:33.351069 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "636257ce6f0f43cdb764aae55a9e764e" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:12:33.351308 30005 raft_consensus.cc:2454] T 636257ce6f0f43cdb764aae55a9e764e P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:33.351286 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "636257ce6f0f43cdb764aae55a9e764e" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:33.351501 29875 raft_consensus.cc:2454] T 636257ce6f0f43cdb764aae55a9e764e P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:33.351693 30200 leader_election.cc:302] T 636257ce6f0f43cdb764aae55a9e764e P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:33.351979 31221 raft_consensus.cc:2781] T 636257ce6f0f43cdb764aae55a9e764e P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:33.352077 31221 raft_consensus.cc:482] T 636257ce6f0f43cdb764aae55a9e764e P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:33.352181 31221 raft_consensus.cc:3037] T 636257ce6f0f43cdb764aae55a9e764e P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.354347 31221 raft_consensus.cc:504] T 636257ce6f0f43cdb764aae55a9e764e P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:33.354921 31221 leader_election.cc:288] T 636257ce6f0f43cdb764aae55a9e764e P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Requested vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:12:33.355362 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "636257ce6f0f43cdb764aae55a9e764e" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:12:33.355549 29875 raft_consensus.cc:3037] T 636257ce6f0f43cdb764aae55a9e764e P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.355486 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "636257ce6f0f43cdb764aae55a9e764e" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:12:33.355675 30005 raft_consensus.cc:3037] T 636257ce6f0f43cdb764aae55a9e764e P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.357510 29875 raft_consensus.cc:2454] T 636257ce6f0f43cdb764aae55a9e764e P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:33.357911 30202 leader_election.cc:302] T 636257ce6f0f43cdb764aae55a9e764e P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:33.358192 31221 raft_consensus.cc:2781] T 636257ce6f0f43cdb764aae55a9e764e P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:33.358366 31221 raft_consensus.cc:686] T 636257ce6f0f43cdb764aae55a9e764e P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Becoming Leader. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Running, Role: LEADER
I20240503 02:12:33.358474 30005 raft_consensus.cc:2454] T 636257ce6f0f43cdb764aae55a9e764e P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:33.358651 31221 consensus_queue.cc:237] T 636257ce6f0f43cdb764aae55a9e764e P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:33.361505 29731 catalog_manager.cc:5484] T 636257ce6f0f43cdb764aae55a9e764e P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4). New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:12:33.363144 31281 raft_consensus.cc:482] T d4a5af9181d0429a8a4f4d734c59e2b6 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:33.363297 31281 raft_consensus.cc:504] T d4a5af9181d0429a8a4f4d734c59e2b6 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:33.363994 31281 leader_election.cc:288] T d4a5af9181d0429a8a4f4d734c59e2b6 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:12:33.365406 30135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4a5af9181d0429a8a4f4d734c59e2b6" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:12:33.365666 30135 raft_consensus.cc:2454] T d4a5af9181d0429a8a4f4d734c59e2b6 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:12:33.365820 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4a5af9181d0429a8a4f4d734c59e2b6" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:12:33.366055 30264 raft_consensus.cc:2454] T d4a5af9181d0429a8a4f4d734c59e2b6 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:12:33.366099 29808 leader_election.cc:302] T d4a5af9181d0429a8a4f4d734c59e2b6 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:33.367272 31281 raft_consensus.cc:2781] T d4a5af9181d0429a8a4f4d734c59e2b6 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:33.367399 31281 raft_consensus.cc:482] T d4a5af9181d0429a8a4f4d734c59e2b6 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:33.367508 31281 raft_consensus.cc:3037] T d4a5af9181d0429a8a4f4d734c59e2b6 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.369539 31281 raft_consensus.cc:504] T d4a5af9181d0429a8a4f4d734c59e2b6 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:33.370110 31281 leader_election.cc:288] T d4a5af9181d0429a8a4f4d734c59e2b6 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Requested vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:12:33.370498 30135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4a5af9181d0429a8a4f4d734c59e2b6" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:12:33.370572 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4a5af9181d0429a8a4f4d734c59e2b6" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:12:33.370707 30135 raft_consensus.cc:3037] T d4a5af9181d0429a8a4f4d734c59e2b6 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.370779 30264 raft_consensus.cc:3037] T d4a5af9181d0429a8a4f4d734c59e2b6 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.373368 30264 raft_consensus.cc:2454] T d4a5af9181d0429a8a4f4d734c59e2b6 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:12:33.373646 30135 raft_consensus.cc:2454] T d4a5af9181d0429a8a4f4d734c59e2b6 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:12:33.373757 29809 leader_election.cc:302] T d4a5af9181d0429a8a4f4d734c59e2b6 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:33.374027 31281 raft_consensus.cc:2781] T d4a5af9181d0429a8a4f4d734c59e2b6 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:33.374167 31281 raft_consensus.cc:686] T d4a5af9181d0429a8a4f4d734c59e2b6 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Becoming Leader. State: Replica: 16440e365430405fb10366ab70e2b429, State: Running, Role: LEADER
I20240503 02:12:33.374410 31281 consensus_queue.cc:237] T d4a5af9181d0429a8a4f4d734c59e2b6 P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:33.377329 29731 catalog_manager.cc:5484] T d4a5af9181d0429a8a4f4d734c59e2b6 P 16440e365430405fb10366ab70e2b429 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16440e365430405fb10366ab70e2b429 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:12:33.717911 31221 consensus_queue.cc:1035] T 8a9e3025d1764ce1aa48071ac3e5d902 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:33.722908 31221 consensus_queue.cc:1035] T 8a9e3025d1764ce1aa48071ac3e5d902 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:33.753557 31393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:33.753929 31393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:33.761389 31393 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240503 02:12:33.821869 31221 consensus_queue.cc:1035] T 636257ce6f0f43cdb764aae55a9e764e P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:33.842200 31388 consensus_queue.cc:1035] T 636257ce6f0f43cdb764aae55a9e764e P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:33.915848 30266 raft_consensus.cc:1261] T d4a5af9181d0429a8a4f4d734c59e2b6 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:33.916568 31281 consensus_queue.cc:1035] T d4a5af9181d0429a8a4f4d734c59e2b6 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:33.924158 30135 raft_consensus.cc:1261] T d4a5af9181d0429a8a4f4d734c59e2b6 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:33.925050 31281 consensus_queue.cc:1035] T d4a5af9181d0429a8a4f4d734c59e2b6 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:34.016683 31398 mvcc.cc:205] Tried to move back new op lower bound from 7023420841622814720 to 7023420839423549440. Current Snapshot: MvccSnapshot[applied={T|T < 7023420841622814720}]
W20240503 02:12:34.044267 30072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.1:36719, user_credentials={real_user=slave}} blocked reactor thread for 69445.4us
INSERT report
rows total: 2000
time total: 265.172 ms
time per row: 0.132586 ms
I20240503 02:12:34.179634 29731 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:34.188660 30114 tablet_service.cc:1467] Processing CreateTablet for tablet b6c9fd3cbbd5453897aa072600db569d (DEFAULT_TABLE table=rebalance_test_table_7 [id=f6554714a9d84d2d864dca86f3ad3642]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:34.188812 29986 tablet_service.cc:1467] Processing CreateTablet for tablet b6c9fd3cbbd5453897aa072600db569d (DEFAULT_TABLE table=rebalance_test_table_7 [id=f6554714a9d84d2d864dca86f3ad3642]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:34.189311 29986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6c9fd3cbbd5453897aa072600db569d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:34.189532 30114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6c9fd3cbbd5453897aa072600db569d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:34.189674 30116 tablet_service.cc:1467] Processing CreateTablet for tablet 0007679ffe8047d79dd8ac778fb6f1ce (DEFAULT_TABLE table=rebalance_test_table_7 [id=f6554714a9d84d2d864dca86f3ad3642]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:34.190023 30116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0007679ffe8047d79dd8ac778fb6f1ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:34.193643 30245 tablet_service.cc:1467] Processing CreateTablet for tablet 2bf8c34ee16d4e0fbc6817d63b2ba52e (DEFAULT_TABLE table=rebalance_test_table_7 [id=f6554714a9d84d2d864dca86f3ad3642]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:34.193879 29984 tablet_service.cc:1467] Processing CreateTablet for tablet 0007679ffe8047d79dd8ac778fb6f1ce (DEFAULT_TABLE table=rebalance_test_table_7 [id=f6554714a9d84d2d864dca86f3ad3642]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:34.193881 30244 tablet_service.cc:1467] Processing CreateTablet for tablet 0007679ffe8047d79dd8ac778fb6f1ce (DEFAULT_TABLE table=rebalance_test_table_7 [id=f6554714a9d84d2d864dca86f3ad3642]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:34.194195 30245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bf8c34ee16d4e0fbc6817d63b2ba52e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:34.194240 29984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0007679ffe8047d79dd8ac778fb6f1ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:34.194872 30115 tablet_service.cc:1467] Processing CreateTablet for tablet 2bf8c34ee16d4e0fbc6817d63b2ba52e (DEFAULT_TABLE table=rebalance_test_table_7 [id=f6554714a9d84d2d864dca86f3ad3642]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:34.195199 30115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bf8c34ee16d4e0fbc6817d63b2ba52e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:34.198565 30244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0007679ffe8047d79dd8ac778fb6f1ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:34.193017 29853 tablet_service.cc:1467] Processing CreateTablet for tablet 2bf8c34ee16d4e0fbc6817d63b2ba52e (DEFAULT_TABLE table=rebalance_test_table_7 [id=f6554714a9d84d2d864dca86f3ad3642]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:34.201233 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bf8c34ee16d4e0fbc6817d63b2ba52e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:34.197445 29855 tablet_service.cc:1467] Processing CreateTablet for tablet b6c9fd3cbbd5453897aa072600db569d (DEFAULT_TABLE table=rebalance_test_table_7 [id=f6554714a9d84d2d864dca86f3ad3642]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:34.205638 29855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6c9fd3cbbd5453897aa072600db569d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:34.215596 31459 tablet_bootstrap.cc:492] T 0007679ffe8047d79dd8ac778fb6f1ce P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:34.219467 31459 tablet_bootstrap.cc:654] T 0007679ffe8047d79dd8ac778fb6f1ce P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.221252 31457 tablet_bootstrap.cc:492] T b6c9fd3cbbd5453897aa072600db569d P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:34.222257 31458 tablet_bootstrap.cc:492] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:34.223834 31458 tablet_bootstrap.cc:654] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.224360 31461 tablet_bootstrap.cc:492] T b6c9fd3cbbd5453897aa072600db569d P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:34.225674 31459 tablet_bootstrap.cc:492] T 0007679ffe8047d79dd8ac778fb6f1ce P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:34.225836 31459 ts_tablet_manager.cc:1367] T 0007679ffe8047d79dd8ac778fb6f1ce P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240503 02:12:34.225868 31461 tablet_bootstrap.cc:654] T b6c9fd3cbbd5453897aa072600db569d P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.226217 31457 tablet_bootstrap.cc:654] T b6c9fd3cbbd5453897aa072600db569d P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.226714 31459 raft_consensus.cc:348] T 0007679ffe8047d79dd8ac778fb6f1ce P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:34.226965 31459 raft_consensus.cc:374] T 0007679ffe8047d79dd8ac778fb6f1ce P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.227061 31459 raft_consensus.cc:729] T 0007679ffe8047d79dd8ac778fb6f1ce P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.227353 31459 consensus_queue.cc:260] T 0007679ffe8047d79dd8ac778fb6f1ce P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:34.228087 31458 tablet_bootstrap.cc:492] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:34.228286 31458 ts_tablet_manager.cc:1367] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 02:12:34.228475 31461 tablet_bootstrap.cc:492] T b6c9fd3cbbd5453897aa072600db569d P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:34.228616 31461 ts_tablet_manager.cc:1367] T b6c9fd3cbbd5453897aa072600db569d P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 02:12:34.229642 31461 raft_consensus.cc:348] T b6c9fd3cbbd5453897aa072600db569d P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:34.229846 31461 raft_consensus.cc:374] T b6c9fd3cbbd5453897aa072600db569d P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.229934 31461 raft_consensus.cc:729] T b6c9fd3cbbd5453897aa072600db569d P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.230160 31461 consensus_queue.cc:260] T b6c9fd3cbbd5453897aa072600db569d P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:34.230608 31459 ts_tablet_manager.cc:1398] T 0007679ffe8047d79dd8ac778fb6f1ce P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:12:34.230971 31459 tablet_bootstrap.cc:492] T b6c9fd3cbbd5453897aa072600db569d P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:34.231635 31461 ts_tablet_manager.cc:1398] T b6c9fd3cbbd5453897aa072600db569d P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:12:34.231921 31461 tablet_bootstrap.cc:492] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:34.233546 31461 tablet_bootstrap.cc:654] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.233893 31459 tablet_bootstrap.cc:654] T b6c9fd3cbbd5453897aa072600db569d P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.234211 31457 tablet_bootstrap.cc:492] T b6c9fd3cbbd5453897aa072600db569d P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:34.234354 31457 ts_tablet_manager.cc:1367] T b6c9fd3cbbd5453897aa072600db569d P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.002s
I20240503 02:12:34.235394 31457 raft_consensus.cc:348] T b6c9fd3cbbd5453897aa072600db569d P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:34.235760 31457 raft_consensus.cc:374] T b6c9fd3cbbd5453897aa072600db569d P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.235929 31457 raft_consensus.cc:729] T b6c9fd3cbbd5453897aa072600db569d P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.236250 31457 consensus_queue.cc:260] T b6c9fd3cbbd5453897aa072600db569d P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:34.237708 31457 ts_tablet_manager.cc:1398] T b6c9fd3cbbd5453897aa072600db569d P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:12:34.238126 31457 tablet_bootstrap.cc:492] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:34.241648 31457 tablet_bootstrap.cc:654] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.244227 31458 raft_consensus.cc:348] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:34.244715 31458 raft_consensus.cc:374] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.244899 31458 raft_consensus.cc:729] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.245762 31457 tablet_bootstrap.cc:492] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:34.246023 31457 ts_tablet_manager.cc:1367] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20240503 02:12:34.246219 31458 consensus_queue.cc:260] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:34.247157 31458 ts_tablet_manager.cc:1398] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240503 02:12:34.247151 31457 raft_consensus.cc:348] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:34.247465 31458 tablet_bootstrap.cc:492] T 0007679ffe8047d79dd8ac778fb6f1ce P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:34.249732 31458 tablet_bootstrap.cc:654] T 0007679ffe8047d79dd8ac778fb6f1ce P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.250027 31457 raft_consensus.cc:374] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.250188 31457 raft_consensus.cc:729] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.252857 31457 consensus_queue.cc:260] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:34.254873 31457 ts_tablet_manager.cc:1398] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 02:12:34.256587 31461 tablet_bootstrap.cc:492] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:34.256906 31461 ts_tablet_manager.cc:1367] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.014s
I20240503 02:12:34.257002 31459 tablet_bootstrap.cc:492] T b6c9fd3cbbd5453897aa072600db569d P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:34.257213 31459 ts_tablet_manager.cc:1367] T b6c9fd3cbbd5453897aa072600db569d P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.018s
I20240503 02:12:34.257992 31459 raft_consensus.cc:348] T b6c9fd3cbbd5453897aa072600db569d P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:34.258055 31461 raft_consensus.cc:348] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:34.258234 31459 raft_consensus.cc:374] T b6c9fd3cbbd5453897aa072600db569d P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.258323 31459 raft_consensus.cc:729] T b6c9fd3cbbd5453897aa072600db569d P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.258358 31461 raft_consensus.cc:374] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.258453 31461 raft_consensus.cc:729] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.258611 31459 consensus_queue.cc:260] T b6c9fd3cbbd5453897aa072600db569d P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:34.258821 31461 consensus_queue.cc:260] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:34.259986 31459 ts_tablet_manager.cc:1398] T b6c9fd3cbbd5453897aa072600db569d P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:12:34.260268 31459 tablet_bootstrap.cc:492] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:34.260306 31461 ts_tablet_manager.cc:1398] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:34.260785 31328 raft_consensus.cc:482] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:34.260941 31328 raft_consensus.cc:504] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:34.261668 31328 leader_election.cc:288] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:34.261891 31459 tablet_bootstrap.cc:654] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.262413 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0007679ffe8047d79dd8ac778fb6f1ce" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:12:34.263108 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0007679ffe8047d79dd8ac778fb6f1ce" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:12:34.263327 30134 raft_consensus.cc:2454] T 0007679ffe8047d79dd8ac778fb6f1ce P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
W20240503 02:12:34.263280 29940 leader_election.cc:341] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:34.264118 29939 leader_election.cc:302] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, a9094240860e43bfb129193f3bfd5be6; no voters: 1f10f80f13c84e80b03eda7085a3b012
I20240503 02:12:34.264580 31328 raft_consensus.cc:2781] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:34.264699 31328 raft_consensus.cc:482] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:34.264787 31328 raft_consensus.cc:3037] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.266940 31328 raft_consensus.cc:504] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:34.267496 31328 leader_election.cc:288] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Requested vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:34.267904 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0007679ffe8047d79dd8ac778fb6f1ce" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:12:34.268152 30134 raft_consensus.cc:3037] T 0007679ffe8047d79dd8ac778fb6f1ce P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.268061 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0007679ffe8047d79dd8ac778fb6f1ce" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
W20240503 02:12:34.268765 29940 leader_election.cc:341] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:34.269564 31458 tablet_bootstrap.cc:492] T 0007679ffe8047d79dd8ac778fb6f1ce P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:34.269730 31458 ts_tablet_manager.cc:1367] T 0007679ffe8047d79dd8ac778fb6f1ce P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.006s
I20240503 02:12:34.270567 31458 raft_consensus.cc:348] T 0007679ffe8047d79dd8ac778fb6f1ce P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:34.270704 31459 tablet_bootstrap.cc:492] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:34.270776 31458 raft_consensus.cc:374] T 0007679ffe8047d79dd8ac778fb6f1ce P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.270841 31459 ts_tablet_manager.cc:1367] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20240503 02:12:34.270860 31458 raft_consensus.cc:729] T 0007679ffe8047d79dd8ac778fb6f1ce P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.271062 30134 raft_consensus.cc:2454] T 0007679ffe8047d79dd8ac778fb6f1ce P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:12:34.271178 31458 consensus_queue.cc:260] T 0007679ffe8047d79dd8ac778fb6f1ce P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:34.271442 29939 leader_election.cc:302] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, a9094240860e43bfb129193f3bfd5be6; no voters: 1f10f80f13c84e80b03eda7085a3b012
I20240503 02:12:34.271581 31459 raft_consensus.cc:348] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:34.271723 31328 raft_consensus.cc:2781] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:34.271771 31459 raft_consensus.cc:374] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.271858 31459 raft_consensus.cc:729] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.272150 31328 raft_consensus.cc:686] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Becoming Leader. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Running, Role: LEADER
I20240503 02:12:34.272087 31459 consensus_queue.cc:260] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:34.272440 31328 consensus_queue.cc:237] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:34.272689 31459 ts_tablet_manager.cc:1398] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:34.274019 31458 ts_tablet_manager.cc:1398] T 0007679ffe8047d79dd8ac778fb6f1ce P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:34.275594 29725 catalog_manager.cc:5484] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: term changed from 0 to 1, leader changed from <none> to 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2). New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:12:34.287374 31435 raft_consensus.cc:482] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:34.287542 31435 raft_consensus.cc:504] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:34.288201 31435 leader_election.cc:288] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:34.288645 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bf8c34ee16d4e0fbc6817d63b2ba52e" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:34.288694 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bf8c34ee16d4e0fbc6817d63b2ba52e" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:12:34.288895 29875 raft_consensus.cc:2454] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:34.288893 30134 raft_consensus.cc:2454] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:34.289299 30199 leader_election.cc:302] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, 1f10f80f13c84e80b03eda7085a3b012, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:34.289553 31435 raft_consensus.cc:2781] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:34.289650 31435 raft_consensus.cc:482] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:34.289714 31435 raft_consensus.cc:3037] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.291491 31435 raft_consensus.cc:504] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:34.292021 31435 leader_election.cc:288] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Requested vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:34.292420 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bf8c34ee16d4e0fbc6817d63b2ba52e" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:12:34.292541 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bf8c34ee16d4e0fbc6817d63b2ba52e" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:12:34.292618 29875 raft_consensus.cc:3037] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.292747 30134 raft_consensus.cc:3037] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.294513 29875 raft_consensus.cc:2454] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:34.294744 30134 raft_consensus.cc:2454] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:34.295127 30199 leader_election.cc:302] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:34.295421 31435 raft_consensus.cc:2781] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:34.295573 31435 raft_consensus.cc:686] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Becoming Leader. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Running, Role: LEADER
I20240503 02:12:34.295876 31435 consensus_queue.cc:237] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:34.299189 29732 catalog_manager.cc:5484] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4). New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:12:34.471551 30869 raft_consensus.cc:482] T b6c9fd3cbbd5453897aa072600db569d P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:34.471753 30869 raft_consensus.cc:504] T b6c9fd3cbbd5453897aa072600db569d P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:34.472525 30869 leader_election.cc:288] T b6c9fd3cbbd5453897aa072600db569d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:34.472908 30006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6c9fd3cbbd5453897aa072600db569d" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:12:34.473172 30006 raft_consensus.cc:2454] T b6c9fd3cbbd5453897aa072600db569d P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 0.
I20240503 02:12:34.473351 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6c9fd3cbbd5453897aa072600db569d" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:34.473579 29875 raft_consensus.cc:2454] T b6c9fd3cbbd5453897aa072600db569d P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 0.
I20240503 02:12:34.473644 30070 leader_election.cc:302] T b6c9fd3cbbd5453897aa072600db569d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:34.473958 30869 raft_consensus.cc:2781] T b6c9fd3cbbd5453897aa072600db569d P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:34.474174 30869 raft_consensus.cc:482] T b6c9fd3cbbd5453897aa072600db569d P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:34.474252 30869 raft_consensus.cc:3037] T b6c9fd3cbbd5453897aa072600db569d P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.476076 30869 raft_consensus.cc:504] T b6c9fd3cbbd5453897aa072600db569d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:34.476603 30869 leader_election.cc:288] T b6c9fd3cbbd5453897aa072600db569d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Requested vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:34.477094 30006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6c9fd3cbbd5453897aa072600db569d" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:12:34.477285 30006 raft_consensus.cc:3037] T b6c9fd3cbbd5453897aa072600db569d P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.477263 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6c9fd3cbbd5453897aa072600db569d" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:12:34.477448 29875 raft_consensus.cc:3037] T b6c9fd3cbbd5453897aa072600db569d P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.479249 30006 raft_consensus.cc:2454] T b6c9fd3cbbd5453897aa072600db569d P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 1.
I20240503 02:12:34.479436 29875 raft_consensus.cc:2454] T b6c9fd3cbbd5453897aa072600db569d P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 1.
I20240503 02:12:34.479643 30070 leader_election.cc:302] T b6c9fd3cbbd5453897aa072600db569d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:34.479955 30869 raft_consensus.cc:2781] T b6c9fd3cbbd5453897aa072600db569d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:34.480090 30869 raft_consensus.cc:686] T b6c9fd3cbbd5453897aa072600db569d P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Becoming Leader. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Running, Role: LEADER
I20240503 02:12:34.480340 30869 consensus_queue.cc:237] T b6c9fd3cbbd5453897aa072600db569d P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:34.483259 29732 catalog_manager.cc:5484] T b6c9fd3cbbd5453897aa072600db569d P a9094240860e43bfb129193f3bfd5be6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9094240860e43bfb129193f3bfd5be6 (127.22.108.3). New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:12:34.674201 31388 consensus_queue.cc:1035] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:34.686607 31479 consensus_queue.cc:1035] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:34.704412 31479 raft_consensus.cc:482] T 0007679ffe8047d79dd8ac778fb6f1ce P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:34.704691 31479 raft_consensus.cc:504] T 0007679ffe8047d79dd8ac778fb6f1ce P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:34.705566 31479 leader_election.cc:288] T 0007679ffe8047d79dd8ac778fb6f1ce P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:34.706017 30006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0007679ffe8047d79dd8ac778fb6f1ce" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:12:34.706216 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0007679ffe8047d79dd8ac778fb6f1ce" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:12:34.706538 30134 raft_consensus.cc:2379] T 0007679ffe8047d79dd8ac778fb6f1ce P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1f10f80f13c84e80b03eda7085a3b012 in current term 1: Already voted for candidate 155735c7f8b14997ac4c770aee2a9401 in this term.
I20240503 02:12:34.707067 30199 leader_election.cc:302] T 0007679ffe8047d79dd8ac778fb6f1ce P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012; no voters: 155735c7f8b14997ac4c770aee2a9401, a9094240860e43bfb129193f3bfd5be6
I20240503 02:12:34.707440 31435 raft_consensus.cc:3037] T 0007679ffe8047d79dd8ac778fb6f1ce P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.709415 31435 raft_consensus.cc:2726] T 0007679ffe8047d79dd8ac778fb6f1ce P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:34.806548 31482 consensus_queue.cc:1035] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:34.806882 31328 consensus_queue.cc:1035] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:34.868072 31474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:34.868388 31474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:34.876252 31474 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:12:34.908535 30869 consensus_queue.cc:1035] T b6c9fd3cbbd5453897aa072600db569d P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_7'
I20240503 02:12:34.913584 30869 consensus_queue.cc:1035] T b6c9fd3cbbd5453897aa072600db569d P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 253.695 ms
time per row: 0.126847 ms
I20240503 02:12:35.282445 29725 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:35.291080 29855 tablet_service.cc:1467] Processing CreateTablet for tablet d983c9264e5f475aa6d71d3f7eff758f (DEFAULT_TABLE table=rebalance_test_table_8 [id=67dd4000d7e143e4b042a4aa61eac02c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.291601 29855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d983c9264e5f475aa6d71d3f7eff758f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.291613 30245 tablet_service.cc:1467] Processing CreateTablet for tablet b8a9660c16e34a2098af2ce527810583 (DEFAULT_TABLE table=rebalance_test_table_8 [id=67dd4000d7e143e4b042a4aa61eac02c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.292043 30245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8a9660c16e34a2098af2ce527810583. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.292078 29853 tablet_service.cc:1467] Processing CreateTablet for tablet b8a9660c16e34a2098af2ce527810583 (DEFAULT_TABLE table=rebalance_test_table_8 [id=67dd4000d7e143e4b042a4aa61eac02c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.292434 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8a9660c16e34a2098af2ce527810583. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.297139 29986 tablet_service.cc:1467] Processing CreateTablet for tablet b8a9660c16e34a2098af2ce527810583 (DEFAULT_TABLE table=rebalance_test_table_8 [id=67dd4000d7e143e4b042a4aa61eac02c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.297438 29984 tablet_service.cc:1467] Processing CreateTablet for tablet d983c9264e5f475aa6d71d3f7eff758f (DEFAULT_TABLE table=rebalance_test_table_8 [id=67dd4000d7e143e4b042a4aa61eac02c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.297544 31543 tablet_bootstrap.cc:492] T d983c9264e5f475aa6d71d3f7eff758f P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:35.297614 29986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8a9660c16e34a2098af2ce527810583. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.297811 29984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d983c9264e5f475aa6d71d3f7eff758f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.299238 31543 tablet_bootstrap.cc:654] T d983c9264e5f475aa6d71d3f7eff758f P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:35.302532 30116 tablet_service.cc:1467] Processing CreateTablet for tablet d983c9264e5f475aa6d71d3f7eff758f (DEFAULT_TABLE table=rebalance_test_table_8 [id=67dd4000d7e143e4b042a4aa61eac02c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.303057 30116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d983c9264e5f475aa6d71d3f7eff758f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.305887 29985 tablet_service.cc:1467] Processing CreateTablet for tablet f511ffcfb1d448a89913dbe432bf83ce (DEFAULT_TABLE table=rebalance_test_table_8 [id=67dd4000d7e143e4b042a4aa61eac02c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.305987 30244 tablet_service.cc:1467] Processing CreateTablet for tablet f511ffcfb1d448a89913dbe432bf83ce (DEFAULT_TABLE table=rebalance_test_table_8 [id=67dd4000d7e143e4b042a4aa61eac02c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.306277 29985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f511ffcfb1d448a89913dbe432bf83ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.306342 30244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f511ffcfb1d448a89913dbe432bf83ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.306890 31544 tablet_bootstrap.cc:492] T b8a9660c16e34a2098af2ce527810583 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:35.308766 31544 tablet_bootstrap.cc:654] T b8a9660c16e34a2098af2ce527810583 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:35.309417 30115 tablet_service.cc:1467] Processing CreateTablet for tablet f511ffcfb1d448a89913dbe432bf83ce (DEFAULT_TABLE table=rebalance_test_table_8 [id=67dd4000d7e143e4b042a4aa61eac02c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.309818 30115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f511ffcfb1d448a89913dbe432bf83ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.321055 31548 tablet_bootstrap.cc:492] T f511ffcfb1d448a89913dbe432bf83ce P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:35.324368 31546 tablet_bootstrap.cc:492] T b8a9660c16e34a2098af2ce527810583 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:35.325580 31544 tablet_bootstrap.cc:492] T b8a9660c16e34a2098af2ce527810583 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:35.325809 31544 ts_tablet_manager.cc:1367] T b8a9660c16e34a2098af2ce527810583 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.014s
I20240503 02:12:35.325924 31543 tablet_bootstrap.cc:492] T d983c9264e5f475aa6d71d3f7eff758f P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:35.326076 31543 ts_tablet_manager.cc:1367] T d983c9264e5f475aa6d71d3f7eff758f P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.000s
I20240503 02:12:35.326298 31546 tablet_bootstrap.cc:654] T b8a9660c16e34a2098af2ce527810583 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:35.326655 31544 raft_consensus.cc:348] T b8a9660c16e34a2098af2ce527810583 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:35.326731 31543 raft_consensus.cc:348] T d983c9264e5f475aa6d71d3f7eff758f P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:35.326894 31543 raft_consensus.cc:374] T d983c9264e5f475aa6d71d3f7eff758f P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:35.326897 31544 raft_consensus.cc:374] T b8a9660c16e34a2098af2ce527810583 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:35.326984 31543 raft_consensus.cc:729] T d983c9264e5f475aa6d71d3f7eff758f P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:35.326989 31544 raft_consensus.cc:729] T b8a9660c16e34a2098af2ce527810583 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:35.327216 31544 consensus_queue.cc:260] T b8a9660c16e34a2098af2ce527810583 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:35.327268 31543 consensus_queue.cc:260] T d983c9264e5f475aa6d71d3f7eff758f P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:35.327910 31544 ts_tablet_manager.cc:1398] T b8a9660c16e34a2098af2ce527810583 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:35.328243 31544 tablet_bootstrap.cc:492] T f511ffcfb1d448a89913dbe432bf83ce P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:35.329450 31548 tablet_bootstrap.cc:654] T f511ffcfb1d448a89913dbe432bf83ce P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:35.329545 31543 ts_tablet_manager.cc:1398] T d983c9264e5f475aa6d71d3f7eff758f P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:35.329800 31543 tablet_bootstrap.cc:492] T b8a9660c16e34a2098af2ce527810583 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:35.329946 31544 tablet_bootstrap.cc:654] T f511ffcfb1d448a89913dbe432bf83ce P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:35.331311 31543 tablet_bootstrap.cc:654] T b8a9660c16e34a2098af2ce527810583 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:35.338168 31544 tablet_bootstrap.cc:492] T f511ffcfb1d448a89913dbe432bf83ce P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:35.338397 31544 ts_tablet_manager.cc:1367] T f511ffcfb1d448a89913dbe432bf83ce P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240503 02:12:35.339136 31546 tablet_bootstrap.cc:492] T b8a9660c16e34a2098af2ce527810583 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:35.339164 31544 raft_consensus.cc:348] T f511ffcfb1d448a89913dbe432bf83ce P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:35.339388 31544 raft_consensus.cc:374] T f511ffcfb1d448a89913dbe432bf83ce P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:35.339466 31544 raft_consensus.cc:729] T f511ffcfb1d448a89913dbe432bf83ce P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:35.339515 31546 ts_tablet_manager.cc:1367] T b8a9660c16e34a2098af2ce527810583 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240503 02:12:35.339692 31544 consensus_queue.cc:260] T f511ffcfb1d448a89913dbe432bf83ce P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:35.340157 31544 ts_tablet_manager.cc:1398] T f511ffcfb1d448a89913dbe432bf83ce P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:35.341804 31543 tablet_bootstrap.cc:492] T b8a9660c16e34a2098af2ce527810583 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:35.341964 31543 ts_tablet_manager.cc:1367] T b8a9660c16e34a2098af2ce527810583 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20240503 02:12:35.342716 31543 raft_consensus.cc:348] T b8a9660c16e34a2098af2ce527810583 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:35.342947 31543 raft_consensus.cc:374] T b8a9660c16e34a2098af2ce527810583 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:35.343021 31543 raft_consensus.cc:729] T b8a9660c16e34a2098af2ce527810583 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:35.343221 31546 raft_consensus.cc:348] T b8a9660c16e34a2098af2ce527810583 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:35.343274 31543 consensus_queue.cc:260] T b8a9660c16e34a2098af2ce527810583 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:35.343644 31546 raft_consensus.cc:374] T b8a9660c16e34a2098af2ce527810583 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:35.343847 31543 ts_tablet_manager.cc:1398] T b8a9660c16e34a2098af2ce527810583 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:35.343957 31546 raft_consensus.cc:729] T b8a9660c16e34a2098af2ce527810583 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:35.344285 31546 consensus_queue.cc:260] T b8a9660c16e34a2098af2ce527810583 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:35.345358 31546 ts_tablet_manager.cc:1398] T b8a9660c16e34a2098af2ce527810583 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:12:35.345604 31546 tablet_bootstrap.cc:492] T f511ffcfb1d448a89913dbe432bf83ce P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:35.347182 31548 tablet_bootstrap.cc:492] T f511ffcfb1d448a89913dbe432bf83ce P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:35.347338 31548 ts_tablet_manager.cc:1367] T f511ffcfb1d448a89913dbe432bf83ce P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.005s
I20240503 02:12:35.347854 31546 tablet_bootstrap.cc:654] T f511ffcfb1d448a89913dbe432bf83ce P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:35.348171 31548 raft_consensus.cc:348] T f511ffcfb1d448a89913dbe432bf83ce P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:35.348541 31548 raft_consensus.cc:374] T f511ffcfb1d448a89913dbe432bf83ce P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:35.348631 31548 raft_consensus.cc:729] T f511ffcfb1d448a89913dbe432bf83ce P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:35.348898 31548 consensus_queue.cc:260] T f511ffcfb1d448a89913dbe432bf83ce P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:35.349539 31548 ts_tablet_manager.cc:1398] T f511ffcfb1d448a89913dbe432bf83ce P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:35.349835 31548 tablet_bootstrap.cc:492] T d983c9264e5f475aa6d71d3f7eff758f P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:35.351910 31548 tablet_bootstrap.cc:654] T d983c9264e5f475aa6d71d3f7eff758f P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:35.357071 31546 tablet_bootstrap.cc:492] T f511ffcfb1d448a89913dbe432bf83ce P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:35.357255 31546 ts_tablet_manager.cc:1367] T f511ffcfb1d448a89913dbe432bf83ce P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20240503 02:12:35.357908 31546 raft_consensus.cc:348] T f511ffcfb1d448a89913dbe432bf83ce P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:35.358160 31546 raft_consensus.cc:374] T f511ffcfb1d448a89913dbe432bf83ce P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:35.358237 31546 raft_consensus.cc:729] T f511ffcfb1d448a89913dbe432bf83ce P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:35.358500 31546 consensus_queue.cc:260] T f511ffcfb1d448a89913dbe432bf83ce P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:35.359699 31546 ts_tablet_manager.cc:1398] T f511ffcfb1d448a89913dbe432bf83ce P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:12:35.359987 31546 tablet_bootstrap.cc:492] T d983c9264e5f475aa6d71d3f7eff758f P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:35.361542 31546 tablet_bootstrap.cc:654] T d983c9264e5f475aa6d71d3f7eff758f P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:35.363569 31548 tablet_bootstrap.cc:492] T d983c9264e5f475aa6d71d3f7eff758f P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:35.363766 31548 ts_tablet_manager.cc:1367] T d983c9264e5f475aa6d71d3f7eff758f P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20240503 02:12:35.364580 31548 raft_consensus.cc:348] T d983c9264e5f475aa6d71d3f7eff758f P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:35.364818 31548 raft_consensus.cc:374] T d983c9264e5f475aa6d71d3f7eff758f P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:35.364907 31548 raft_consensus.cc:729] T d983c9264e5f475aa6d71d3f7eff758f P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:35.365193 31548 consensus_queue.cc:260] T d983c9264e5f475aa6d71d3f7eff758f P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:35.365832 31548 ts_tablet_manager.cc:1398] T d983c9264e5f475aa6d71d3f7eff758f P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:35.366181 31546 tablet_bootstrap.cc:492] T d983c9264e5f475aa6d71d3f7eff758f P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:35.366310 31546 ts_tablet_manager.cc:1367] T d983c9264e5f475aa6d71d3f7eff758f P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:12:35.367017 31546 raft_consensus.cc:348] T d983c9264e5f475aa6d71d3f7eff758f P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:35.367237 31546 raft_consensus.cc:374] T d983c9264e5f475aa6d71d3f7eff758f P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:35.367336 31546 raft_consensus.cc:729] T d983c9264e5f475aa6d71d3f7eff758f P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:35.367570 31546 consensus_queue.cc:260] T d983c9264e5f475aa6d71d3f7eff758f P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:35.368098 31546 ts_tablet_manager.cc:1398] T d983c9264e5f475aa6d71d3f7eff758f P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:35.375427 31435 raft_consensus.cc:482] T f511ffcfb1d448a89913dbe432bf83ce P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:35.375619 31435 raft_consensus.cc:504] T f511ffcfb1d448a89913dbe432bf83ce P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:35.376294 31435 leader_election.cc:288] T f511ffcfb1d448a89913dbe432bf83ce P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:35.376680 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f511ffcfb1d448a89913dbe432bf83ce" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:12:35.376866 30005 raft_consensus.cc:2454] T f511ffcfb1d448a89913dbe432bf83ce P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:35.376793 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f511ffcfb1d448a89913dbe432bf83ce" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:12:35.377009 30134 raft_consensus.cc:2454] T f511ffcfb1d448a89913dbe432bf83ce P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:35.377254 30200 leader_election.cc:302] T f511ffcfb1d448a89913dbe432bf83ce P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:35.377563 31435 raft_consensus.cc:2781] T f511ffcfb1d448a89913dbe432bf83ce P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:35.377657 31435 raft_consensus.cc:482] T f511ffcfb1d448a89913dbe432bf83ce P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:35.377765 31435 raft_consensus.cc:3037] T f511ffcfb1d448a89913dbe432bf83ce P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.380575 31435 raft_consensus.cc:504] T f511ffcfb1d448a89913dbe432bf83ce P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:35.381196 31435 leader_election.cc:288] T f511ffcfb1d448a89913dbe432bf83ce P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Requested vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:35.381763 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f511ffcfb1d448a89913dbe432bf83ce" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:12:35.382002 30005 raft_consensus.cc:3037] T f511ffcfb1d448a89913dbe432bf83ce P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.381971 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f511ffcfb1d448a89913dbe432bf83ce" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:12:35.382159 30134 raft_consensus.cc:3037] T f511ffcfb1d448a89913dbe432bf83ce P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.383819 30005 raft_consensus.cc:2454] T f511ffcfb1d448a89913dbe432bf83ce P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:35.384186 30200 leader_election.cc:302] T f511ffcfb1d448a89913dbe432bf83ce P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:35.384438 31435 raft_consensus.cc:2781] T f511ffcfb1d448a89913dbe432bf83ce P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:35.384609 31435 raft_consensus.cc:686] T f511ffcfb1d448a89913dbe432bf83ce P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Becoming Leader. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Running, Role: LEADER
I20240503 02:12:35.384904 30134 raft_consensus.cc:2454] T f511ffcfb1d448a89913dbe432bf83ce P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:35.385160 31435 consensus_queue.cc:237] T f511ffcfb1d448a89913dbe432bf83ce P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:35.387971 29732 catalog_manager.cc:5484] T f511ffcfb1d448a89913dbe432bf83ce P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4). New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:12:35.417236 31435 raft_consensus.cc:482] T b8a9660c16e34a2098af2ce527810583 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:35.417403 31435 raft_consensus.cc:504] T b8a9660c16e34a2098af2ce527810583 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:35.418020 31435 leader_election.cc:288] T b8a9660c16e34a2098af2ce527810583 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:35.418540 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a9660c16e34a2098af2ce527810583" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:12:35.418540 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a9660c16e34a2098af2ce527810583" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:35.418789 30005 raft_consensus.cc:2454] T b8a9660c16e34a2098af2ce527810583 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:35.418797 29875 raft_consensus.cc:2454] T b8a9660c16e34a2098af2ce527810583 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:35.419162 30200 leader_election.cc:302] T b8a9660c16e34a2098af2ce527810583 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:35.419529 31435 raft_consensus.cc:2781] T b8a9660c16e34a2098af2ce527810583 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:35.419641 31435 raft_consensus.cc:482] T b8a9660c16e34a2098af2ce527810583 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:35.419730 31435 raft_consensus.cc:3037] T b8a9660c16e34a2098af2ce527810583 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.421875 31435 raft_consensus.cc:504] T b8a9660c16e34a2098af2ce527810583 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:35.422421 31435 leader_election.cc:288] T b8a9660c16e34a2098af2ce527810583 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Requested vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:35.422781 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a9660c16e34a2098af2ce527810583" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:12:35.422993 30005 raft_consensus.cc:3037] T b8a9660c16e34a2098af2ce527810583 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.423028 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a9660c16e34a2098af2ce527810583" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:12:35.423251 29875 raft_consensus.cc:3037] T b8a9660c16e34a2098af2ce527810583 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.425825 30005 raft_consensus.cc:2454] T b8a9660c16e34a2098af2ce527810583 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:35.425894 29875 raft_consensus.cc:2454] T b8a9660c16e34a2098af2ce527810583 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:35.426285 30200 leader_election.cc:302] T b8a9660c16e34a2098af2ce527810583 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:35.426581 31435 raft_consensus.cc:2781] T b8a9660c16e34a2098af2ce527810583 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:35.426735 31435 raft_consensus.cc:686] T b8a9660c16e34a2098af2ce527810583 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Becoming Leader. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Running, Role: LEADER
I20240503 02:12:35.427027 31435 consensus_queue.cc:237] T b8a9660c16e34a2098af2ce527810583 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:35.430099 29732 catalog_manager.cc:5484] T b8a9660c16e34a2098af2ce527810583 P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4). New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:12:35.463616 31529 raft_consensus.cc:482] T d983c9264e5f475aa6d71d3f7eff758f P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:35.463863 31529 raft_consensus.cc:504] T d983c9264e5f475aa6d71d3f7eff758f P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:35.464780 31529 leader_election.cc:288] T d983c9264e5f475aa6d71d3f7eff758f P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:35.465193 30006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d983c9264e5f475aa6d71d3f7eff758f" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:12:35.465467 30006 raft_consensus.cc:2454] T d983c9264e5f475aa6d71d3f7eff758f P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 0.
I20240503 02:12:35.465888 30070 leader_election.cc:302] T d983c9264e5f475aa6d71d3f7eff758f P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:35.465929 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d983c9264e5f475aa6d71d3f7eff758f" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:35.466166 29875 raft_consensus.cc:2454] T d983c9264e5f475aa6d71d3f7eff758f P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 0.
I20240503 02:12:35.466229 31529 raft_consensus.cc:2781] T d983c9264e5f475aa6d71d3f7eff758f P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:35.466353 31529 raft_consensus.cc:482] T d983c9264e5f475aa6d71d3f7eff758f P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:35.466449 31529 raft_consensus.cc:3037] T d983c9264e5f475aa6d71d3f7eff758f P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.468628 31529 raft_consensus.cc:504] T d983c9264e5f475aa6d71d3f7eff758f P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:35.469167 31529 leader_election.cc:288] T d983c9264e5f475aa6d71d3f7eff758f P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Requested vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:35.469550 30006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d983c9264e5f475aa6d71d3f7eff758f" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:12:35.469691 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d983c9264e5f475aa6d71d3f7eff758f" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:12:35.469805 30006 raft_consensus.cc:3037] T d983c9264e5f475aa6d71d3f7eff758f P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.469887 29875 raft_consensus.cc:3037] T d983c9264e5f475aa6d71d3f7eff758f P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.472651 30006 raft_consensus.cc:2454] T d983c9264e5f475aa6d71d3f7eff758f P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 1.
I20240503 02:12:35.472667 29875 raft_consensus.cc:2454] T d983c9264e5f475aa6d71d3f7eff758f P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 1.
I20240503 02:12:35.473146 30072 leader_election.cc:302] T d983c9264e5f475aa6d71d3f7eff758f P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:35.473481 31529 raft_consensus.cc:2781] T d983c9264e5f475aa6d71d3f7eff758f P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:35.473608 31529 raft_consensus.cc:686] T d983c9264e5f475aa6d71d3f7eff758f P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Becoming Leader. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Running, Role: LEADER
I20240503 02:12:35.473896 31529 consensus_queue.cc:237] T d983c9264e5f475aa6d71d3f7eff758f P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:35.476668 29732 catalog_manager.cc:5484] T d983c9264e5f475aa6d71d3f7eff758f P a9094240860e43bfb129193f3bfd5be6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9094240860e43bfb129193f3bfd5be6 (127.22.108.3). New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:12:35.796240 31435 consensus_queue.cc:1035] T f511ffcfb1d448a89913dbe432bf83ce P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:35.827250 31435 consensus_queue.cc:1035] T f511ffcfb1d448a89913dbe432bf83ce P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:35.847677 31435 consensus_queue.cc:1035] T b8a9660c16e34a2098af2ce527810583 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:35.863018 31435 consensus_queue.cc:1035] T b8a9660c16e34a2098af2ce527810583 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:35.866389 31559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:35.867054 31559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:35.881131 31559 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:12:35.891974 31529 consensus_queue.cc:1035] T d983c9264e5f475aa6d71d3f7eff758f P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:35.899485 31440 consensus_queue.cc:1035] T d983c9264e5f475aa6d71d3f7eff758f P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_8'
INSERT report
rows total: 2000
time total: 306.927 ms
time per row: 0.153464 ms
I20240503 02:12:36.344703 29734 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:36.354305 30244 tablet_service.cc:1467] Processing CreateTablet for tablet df37d88053e2419fbe40d03023d0c756 (DEFAULT_TABLE table=rebalance_test_table_9 [id=e80de094521a4d71bc2fb5f3343275b8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.354811 30244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df37d88053e2419fbe40d03023d0c756. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.355011 30245 tablet_service.cc:1467] Processing CreateTablet for tablet fa288463ae154e28afd9c71b4b55ef3a (DEFAULT_TABLE table=rebalance_test_table_9 [id=e80de094521a4d71bc2fb5f3343275b8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.355438 30245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa288463ae154e28afd9c71b4b55ef3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.356448 29853 tablet_service.cc:1467] Processing CreateTablet for tablet df37d88053e2419fbe40d03023d0c756 (DEFAULT_TABLE table=rebalance_test_table_9 [id=e80de094521a4d71bc2fb5f3343275b8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.356873 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df37d88053e2419fbe40d03023d0c756. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.357473 30114 tablet_service.cc:1467] Processing CreateTablet for tablet 7076453cbe5944eb9b075127e1e0495d (DEFAULT_TABLE table=rebalance_test_table_9 [id=e80de094521a4d71bc2fb5f3343275b8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.357896 30114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7076453cbe5944eb9b075127e1e0495d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.357964 29855 tablet_service.cc:1467] Processing CreateTablet for tablet fa288463ae154e28afd9c71b4b55ef3a (DEFAULT_TABLE table=rebalance_test_table_9 [id=e80de094521a4d71bc2fb5f3343275b8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.358338 29855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa288463ae154e28afd9c71b4b55ef3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.362977 29854 tablet_service.cc:1467] Processing CreateTablet for tablet 7076453cbe5944eb9b075127e1e0495d (DEFAULT_TABLE table=rebalance_test_table_9 [id=e80de094521a4d71bc2fb5f3343275b8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.363404 29854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7076453cbe5944eb9b075127e1e0495d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.364197 30115 tablet_service.cc:1467] Processing CreateTablet for tablet fa288463ae154e28afd9c71b4b55ef3a (DEFAULT_TABLE table=rebalance_test_table_9 [id=e80de094521a4d71bc2fb5f3343275b8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.364235 30116 tablet_service.cc:1467] Processing CreateTablet for tablet df37d88053e2419fbe40d03023d0c756 (DEFAULT_TABLE table=rebalance_test_table_9 [id=e80de094521a4d71bc2fb5f3343275b8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.364938 29986 tablet_service.cc:1467] Processing CreateTablet for tablet 7076453cbe5944eb9b075127e1e0495d (DEFAULT_TABLE table=rebalance_test_table_9 [id=e80de094521a4d71bc2fb5f3343275b8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.365413 29986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7076453cbe5944eb9b075127e1e0495d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.368899 30115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa288463ae154e28afd9c71b4b55ef3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.369478 30116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df37d88053e2419fbe40d03023d0c756. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.379215 31628 tablet_bootstrap.cc:492] T 7076453cbe5944eb9b075127e1e0495d P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:36.381639 31626 tablet_bootstrap.cc:492] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:36.383584 31628 tablet_bootstrap.cc:654] T 7076453cbe5944eb9b075127e1e0495d P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:36.387609 31626 tablet_bootstrap.cc:654] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:36.391294 31627 tablet_bootstrap.cc:492] T df37d88053e2419fbe40d03023d0c756 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:36.393841 31627 tablet_bootstrap.cc:654] T df37d88053e2419fbe40d03023d0c756 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:36.396169 31630 tablet_bootstrap.cc:492] T df37d88053e2419fbe40d03023d0c756 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:36.396950 31628 tablet_bootstrap.cc:492] T 7076453cbe5944eb9b075127e1e0495d P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:36.397549 31628 ts_tablet_manager.cc:1367] T 7076453cbe5944eb9b075127e1e0495d P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.005s
I20240503 02:12:36.398147 31630 tablet_bootstrap.cc:654] T df37d88053e2419fbe40d03023d0c756 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:36.398607 31628 raft_consensus.cc:348] T 7076453cbe5944eb9b075127e1e0495d P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:36.398938 31628 raft_consensus.cc:374] T 7076453cbe5944eb9b075127e1e0495d P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:36.399075 31628 raft_consensus.cc:729] T 7076453cbe5944eb9b075127e1e0495d P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:36.399412 31628 consensus_queue.cc:260] T 7076453cbe5944eb9b075127e1e0495d P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:36.401702 31628 ts_tablet_manager.cc:1398] T 7076453cbe5944eb9b075127e1e0495d P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:12:36.403190 31626 tablet_bootstrap.cc:492] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:36.403703 31626 ts_tablet_manager.cc:1367] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.000s
I20240503 02:12:36.405571 31626 raft_consensus.cc:348] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:36.405879 31626 raft_consensus.cc:374] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:36.406000 31626 raft_consensus.cc:729] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:36.406229 31626 consensus_queue.cc:260] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:36.408188 31626 ts_tablet_manager.cc:1398] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:12:36.408468 31626 tablet_bootstrap.cc:492] T df37d88053e2419fbe40d03023d0c756 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:36.409912 31626 tablet_bootstrap.cc:654] T df37d88053e2419fbe40d03023d0c756 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:36.411445 31627 tablet_bootstrap.cc:492] T df37d88053e2419fbe40d03023d0c756 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:36.411597 31627 ts_tablet_manager.cc:1367] T df37d88053e2419fbe40d03023d0c756 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.004s
I20240503 02:12:36.412209 31627 raft_consensus.cc:348] T df37d88053e2419fbe40d03023d0c756 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:36.412381 31627 raft_consensus.cc:374] T df37d88053e2419fbe40d03023d0c756 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:36.412474 31627 raft_consensus.cc:729] T df37d88053e2419fbe40d03023d0c756 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:36.412685 31627 consensus_queue.cc:260] T df37d88053e2419fbe40d03023d0c756 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:36.413254 31627 ts_tablet_manager.cc:1398] T df37d88053e2419fbe40d03023d0c756 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:36.414217 31627 tablet_bootstrap.cc:492] T fa288463ae154e28afd9c71b4b55ef3a P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:36.415808 31627 tablet_bootstrap.cc:654] T fa288463ae154e28afd9c71b4b55ef3a P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:36.421250 31627 tablet_bootstrap.cc:492] T fa288463ae154e28afd9c71b4b55ef3a P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:36.421433 31627 ts_tablet_manager.cc:1367] T fa288463ae154e28afd9c71b4b55ef3a P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 02:12:36.422185 31630 tablet_bootstrap.cc:492] T df37d88053e2419fbe40d03023d0c756 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:36.422407 31630 ts_tablet_manager.cc:1367] T df37d88053e2419fbe40d03023d0c756 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.020s
I20240503 02:12:36.422376 31627 raft_consensus.cc:348] T fa288463ae154e28afd9c71b4b55ef3a P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:36.422672 31627 raft_consensus.cc:374] T fa288463ae154e28afd9c71b4b55ef3a P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:36.422777 31627 raft_consensus.cc:729] T fa288463ae154e28afd9c71b4b55ef3a P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:36.423043 31627 consensus_queue.cc:260] T fa288463ae154e28afd9c71b4b55ef3a P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:36.423354 31626 tablet_bootstrap.cc:492] T df37d88053e2419fbe40d03023d0c756 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:36.423527 31626 ts_tablet_manager.cc:1367] T df37d88053e2419fbe40d03023d0c756 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.003s
I20240503 02:12:36.423544 31630 raft_consensus.cc:348] T df37d88053e2419fbe40d03023d0c756 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:36.423763 31630 raft_consensus.cc:374] T df37d88053e2419fbe40d03023d0c756 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:36.423733 31435 raft_consensus.cc:482] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:36.423849 31630 raft_consensus.cc:729] T df37d88053e2419fbe40d03023d0c756 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:36.423921 31435 raft_consensus.cc:504] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:36.424101 31630 consensus_queue.cc:260] T df37d88053e2419fbe40d03023d0c756 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:36.424441 31626 raft_consensus.cc:348] T df37d88053e2419fbe40d03023d0c756 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:36.424639 31626 raft_consensus.cc:374] T df37d88053e2419fbe40d03023d0c756 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:36.424667 31435 leader_election.cc:288] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:36.424705 31626 raft_consensus.cc:729] T df37d88053e2419fbe40d03023d0c756 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:36.424789 31630 ts_tablet_manager.cc:1398] T df37d88053e2419fbe40d03023d0c756 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:12:36.424911 31626 consensus_queue.cc:260] T df37d88053e2419fbe40d03023d0c756 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:36.425089 31630 tablet_bootstrap.cc:492] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:36.425426 31626 ts_tablet_manager.cc:1398] T df37d88053e2419fbe40d03023d0c756 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:12:36.426946 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa288463ae154e28afd9c71b4b55ef3a" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:12:36.427460 31630 tablet_bootstrap.cc:654] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:36.429210 30199 leader_election.cc:341] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:36.430135 29874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa288463ae154e28afd9c71b4b55ef3a" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:36.430330 29874 raft_consensus.cc:2454] T fa288463ae154e28afd9c71b4b55ef3a P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:36.431068 30202 leader_election.cc:302] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16440e365430405fb10366ab70e2b429, 1f10f80f13c84e80b03eda7085a3b012; no voters: a9094240860e43bfb129193f3bfd5be6
I20240503 02:12:36.432467 31435 raft_consensus.cc:2781] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:36.432585 31435 raft_consensus.cc:482] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:36.432700 31435 raft_consensus.cc:3037] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:36.434509 31627 ts_tablet_manager.cc:1398] T fa288463ae154e28afd9c71b4b55ef3a P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240503 02:12:36.434968 31627 tablet_bootstrap.cc:492] T 7076453cbe5944eb9b075127e1e0495d P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:36.436165 31435 raft_consensus.cc:504] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:36.437390 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa288463ae154e28afd9c71b4b55ef3a" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:12:36.438058 31627 tablet_bootstrap.cc:654] T 7076453cbe5944eb9b075127e1e0495d P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:36.438107 30199 leader_election.cc:341] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:36.442988 29874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa288463ae154e28afd9c71b4b55ef3a" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:12:36.443189 29874 raft_consensus.cc:3037] T fa288463ae154e28afd9c71b4b55ef3a P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:36.447387 29874 raft_consensus.cc:2454] T fa288463ae154e28afd9c71b4b55ef3a P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:36.447816 30202 leader_election.cc:302] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16440e365430405fb10366ab70e2b429, 1f10f80f13c84e80b03eda7085a3b012; no voters: a9094240860e43bfb129193f3bfd5be6
I20240503 02:12:36.448143 31388 raft_consensus.cc:2781] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:36.448297 31388 raft_consensus.cc:686] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Becoming Leader. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Running, Role: LEADER
I20240503 02:12:36.448666 31388 consensus_queue.cc:237] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:36.450340 31435 leader_election.cc:288] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Requested vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:36.451041 31627 tablet_bootstrap.cc:492] T 7076453cbe5944eb9b075127e1e0495d P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:36.451212 31627 ts_tablet_manager.cc:1367] T 7076453cbe5944eb9b075127e1e0495d P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.007s
I20240503 02:12:36.452775 31627 raft_consensus.cc:348] T 7076453cbe5944eb9b075127e1e0495d P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:36.453092 31627 raft_consensus.cc:374] T 7076453cbe5944eb9b075127e1e0495d P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:36.453227 31627 raft_consensus.cc:729] T 7076453cbe5944eb9b075127e1e0495d P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:36.453522 31627 consensus_queue.cc:260] T 7076453cbe5944eb9b075127e1e0495d P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:36.455299 31630 tablet_bootstrap.cc:492] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:36.455488 31630 ts_tablet_manager.cc:1367] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.030s user 0.019s sys 0.005s
I20240503 02:12:36.456111 31627 ts_tablet_manager.cc:1398] T 7076453cbe5944eb9b075127e1e0495d P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20240503 02:12:36.456118 29731 catalog_manager.cc:5484] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4). New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:12:36.456744 31630 raft_consensus.cc:348] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:36.456933 31630 raft_consensus.cc:374] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:36.459553 31630 raft_consensus.cc:729] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:36.459861 31630 consensus_queue.cc:260] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:36.460474 31630 ts_tablet_manager.cc:1398] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 02:12:36.460793 31630 tablet_bootstrap.cc:492] T fa288463ae154e28afd9c71b4b55ef3a P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:36.462420 31630 tablet_bootstrap.cc:654] T fa288463ae154e28afd9c71b4b55ef3a P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:36.465586 31630 tablet_bootstrap.cc:492] T fa288463ae154e28afd9c71b4b55ef3a P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:36.465757 31630 ts_tablet_manager.cc:1367] T fa288463ae154e28afd9c71b4b55ef3a P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 02:12:36.466428 31630 raft_consensus.cc:348] T fa288463ae154e28afd9c71b4b55ef3a P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:36.466583 31630 raft_consensus.cc:374] T fa288463ae154e28afd9c71b4b55ef3a P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:36.466648 31630 raft_consensus.cc:729] T fa288463ae154e28afd9c71b4b55ef3a P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:36.466809 31630 consensus_queue.cc:260] T fa288463ae154e28afd9c71b4b55ef3a P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:36.467244 31630 ts_tablet_manager.cc:1398] T fa288463ae154e28afd9c71b4b55ef3a P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:12:36.516263 31440 raft_consensus.cc:482] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:36.516494 31440 raft_consensus.cc:504] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:36.517244 31440 leader_election.cc:288] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:12:36.517736 29874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7076453cbe5944eb9b075127e1e0495d" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:36.517802 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7076453cbe5944eb9b075127e1e0495d" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:12:36.518033 30005 raft_consensus.cc:2454] T 7076453cbe5944eb9b075127e1e0495d P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 0.
I20240503 02:12:36.518033 29874 raft_consensus.cc:2454] T 7076453cbe5944eb9b075127e1e0495d P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 0.
I20240503 02:12:36.518500 30070 leader_election.cc:302] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:36.518843 31440 raft_consensus.cc:2781] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:36.518934 31440 raft_consensus.cc:482] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:36.519057 31440 raft_consensus.cc:3037] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:36.520985 31440 raft_consensus.cc:504] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:36.521638 31440 leader_election.cc:288] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Requested vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:12:36.522083 29874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7076453cbe5944eb9b075127e1e0495d" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:12:36.522122 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7076453cbe5944eb9b075127e1e0495d" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:12:36.522311 29874 raft_consensus.cc:3037] T 7076453cbe5944eb9b075127e1e0495d P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:36.522327 30005 raft_consensus.cc:3037] T 7076453cbe5944eb9b075127e1e0495d P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:36.525211 30005 raft_consensus.cc:2454] T 7076453cbe5944eb9b075127e1e0495d P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 1.
I20240503 02:12:36.525211 29874 raft_consensus.cc:2454] T 7076453cbe5944eb9b075127e1e0495d P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 1.
I20240503 02:12:36.525624 30072 leader_election.cc:302] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:36.525951 31440 raft_consensus.cc:2781] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:36.526226 31440 raft_consensus.cc:686] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Becoming Leader. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Running, Role: LEADER
I20240503 02:12:36.526484 31440 consensus_queue.cc:237] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:36.529316 29731 catalog_manager.cc:5484] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9094240860e43bfb129193f3bfd5be6 (127.22.108.3). New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:12:36.540218 31475 raft_consensus.cc:482] T df37d88053e2419fbe40d03023d0c756 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:36.540407 31475 raft_consensus.cc:504] T df37d88053e2419fbe40d03023d0c756 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:36.541126 31475 leader_election.cc:288] T df37d88053e2419fbe40d03023d0c756 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:36.541571 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df37d88053e2419fbe40d03023d0c756" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:12:36.541731 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df37d88053e2419fbe40d03023d0c756" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:12:36.541807 30264 raft_consensus.cc:2454] T df37d88053e2419fbe40d03023d0c756 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:12:36.541954 30134 raft_consensus.cc:2454] T df37d88053e2419fbe40d03023d0c756 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:12:36.542256 29809 leader_election.cc:302] T df37d88053e2419fbe40d03023d0c756 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:36.542555 31475 raft_consensus.cc:2781] T df37d88053e2419fbe40d03023d0c756 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:36.542663 31475 raft_consensus.cc:482] T df37d88053e2419fbe40d03023d0c756 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:36.542737 31475 raft_consensus.cc:3037] T df37d88053e2419fbe40d03023d0c756 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:36.544662 31475 raft_consensus.cc:504] T df37d88053e2419fbe40d03023d0c756 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:36.545251 31475 leader_election.cc:288] T df37d88053e2419fbe40d03023d0c756 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Requested vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:36.545734 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df37d88053e2419fbe40d03023d0c756" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:12:36.545779 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df37d88053e2419fbe40d03023d0c756" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:12:36.545955 30134 raft_consensus.cc:3037] T df37d88053e2419fbe40d03023d0c756 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:36.545977 30264 raft_consensus.cc:3037] T df37d88053e2419fbe40d03023d0c756 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:36.548792 30264 raft_consensus.cc:2454] T df37d88053e2419fbe40d03023d0c756 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:12:36.548805 30134 raft_consensus.cc:2454] T df37d88053e2419fbe40d03023d0c756 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:12:36.549275 29808 leader_election.cc:302] T df37d88053e2419fbe40d03023d0c756 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:36.549571 31475 raft_consensus.cc:2781] T df37d88053e2419fbe40d03023d0c756 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:36.549968 31475 raft_consensus.cc:686] T df37d88053e2419fbe40d03023d0c756 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Becoming Leader. State: Replica: 16440e365430405fb10366ab70e2b429, State: Running, Role: LEADER
I20240503 02:12:36.550330 31475 consensus_queue.cc:237] T df37d88053e2419fbe40d03023d0c756 P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:36.553088 29731 catalog_manager.cc:5484] T df37d88053e2419fbe40d03023d0c756 P 16440e365430405fb10366ab70e2b429 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16440e365430405fb10366ab70e2b429 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:12:36.778371 31440 raft_consensus.cc:482] T fa288463ae154e28afd9c71b4b55ef3a P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:36.778597 31440 raft_consensus.cc:504] T fa288463ae154e28afd9c71b4b55ef3a P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:36.779310 31440 leader_election.cc:288] T fa288463ae154e28afd9c71b4b55ef3a P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:36.779913 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa288463ae154e28afd9c71b4b55ef3a" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:12:36.780073 29874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa288463ae154e28afd9c71b4b55ef3a" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:36.780375 29874 raft_consensus.cc:2379] T fa288463ae154e28afd9c71b4b55ef3a P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a9094240860e43bfb129193f3bfd5be6 in current term 1: Already voted for candidate 1f10f80f13c84e80b03eda7085a3b012 in this term.
I20240503 02:12:36.780800 30072 leader_election.cc:302] T fa288463ae154e28afd9c71b4b55ef3a P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6; no voters: 16440e365430405fb10366ab70e2b429, 1f10f80f13c84e80b03eda7085a3b012
I20240503 02:12:36.781302 31440 raft_consensus.cc:3037] T fa288463ae154e28afd9c71b4b55ef3a P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:36.783428 31440 raft_consensus.cc:2726] T fa288463ae154e28afd9c71b4b55ef3a P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:36.880705 31388 consensus_queue.cc:1035] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:36.890249 31435 consensus_queue.cc:1035] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:36.935243 31475 consensus_queue.cc:1035] T df37d88053e2419fbe40d03023d0c756 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:36.935739 31440 consensus_queue.cc:1035] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:36.941251 31604 consensus_queue.cc:1035] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:36.944279 31641 consensus_queue.cc:1035] T df37d88053e2419fbe40d03023d0c756 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:36.954210 31643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:36.954742 31643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:36.964709 31643 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
INSERT report
rows total: 2000
time total: 203.847 ms
time per row: 0.101923 ms
W20240503 02:12:37.236999 30200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.2:37873, user_credentials={real_user=slave}} blocked reactor thread for 51061.8us
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 02:12:38.292964 29732 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:38.304023 29986 tablet_service.cc:1467] Processing CreateTablet for tablet ad7b19f47b464333ad4c2736019c8fd6 (DEFAULT_TABLE table=rebalance_test_table_10 [id=60b67e8bde6c4177a72606ad2be64acf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.304143 29853 tablet_service.cc:1467] Processing CreateTablet for tablet ad7b19f47b464333ad4c2736019c8fd6 (DEFAULT_TABLE table=rebalance_test_table_10 [id=60b67e8bde6c4177a72606ad2be64acf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.304543 29986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad7b19f47b464333ad4c2736019c8fd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.304612 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad7b19f47b464333ad4c2736019c8fd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.305079 30245 tablet_service.cc:1467] Processing CreateTablet for tablet ad7b19f47b464333ad4c2736019c8fd6 (DEFAULT_TABLE table=rebalance_test_table_10 [id=60b67e8bde6c4177a72606ad2be64acf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.305141 29984 tablet_service.cc:1467] Processing CreateTablet for tablet aef29abf9b4e4cc195ad65ae15b00fd5 (DEFAULT_TABLE table=rebalance_test_table_10 [id=60b67e8bde6c4177a72606ad2be64acf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.305473 30245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad7b19f47b464333ad4c2736019c8fd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.305493 29984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aef29abf9b4e4cc195ad65ae15b00fd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.311174 30116 tablet_service.cc:1467] Processing CreateTablet for tablet aef29abf9b4e4cc195ad65ae15b00fd5 (DEFAULT_TABLE table=rebalance_test_table_10 [id=60b67e8bde6c4177a72606ad2be64acf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.311619 30116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aef29abf9b4e4cc195ad65ae15b00fd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.311950 31714 tablet_bootstrap.cc:492] T ad7b19f47b464333ad4c2736019c8fd6 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:38.313717 31714 tablet_bootstrap.cc:654] T ad7b19f47b464333ad4c2736019c8fd6 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:38.317680 29985 tablet_service.cc:1467] Processing CreateTablet for tablet cb7965d0d50944c6be98c27f40f46de7 (DEFAULT_TABLE table=rebalance_test_table_10 [id=60b67e8bde6c4177a72606ad2be64acf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.318212 29985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb7965d0d50944c6be98c27f40f46de7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.318764 31716 tablet_bootstrap.cc:492] T aef29abf9b4e4cc195ad65ae15b00fd5 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:38.319316 31715 tablet_bootstrap.cc:492] T ad7b19f47b464333ad4c2736019c8fd6 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:38.321197 31715 tablet_bootstrap.cc:654] T ad7b19f47b464333ad4c2736019c8fd6 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:38.321583 30244 tablet_service.cc:1467] Processing CreateTablet for tablet aef29abf9b4e4cc195ad65ae15b00fd5 (DEFAULT_TABLE table=rebalance_test_table_10 [id=60b67e8bde6c4177a72606ad2be64acf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.321738 31716 tablet_bootstrap.cc:654] T aef29abf9b4e4cc195ad65ae15b00fd5 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:38.322016 30244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aef29abf9b4e4cc195ad65ae15b00fd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.325587 30246 tablet_service.cc:1467] Processing CreateTablet for tablet cb7965d0d50944c6be98c27f40f46de7 (DEFAULT_TABLE table=rebalance_test_table_10 [id=60b67e8bde6c4177a72606ad2be64acf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.326104 30246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb7965d0d50944c6be98c27f40f46de7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.329416 31715 tablet_bootstrap.cc:492] T ad7b19f47b464333ad4c2736019c8fd6 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:38.329596 31715 ts_tablet_manager.cc:1367] T ad7b19f47b464333ad4c2736019c8fd6 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20240503 02:12:38.330253 29854 tablet_service.cc:1467] Processing CreateTablet for tablet cb7965d0d50944c6be98c27f40f46de7 (DEFAULT_TABLE table=rebalance_test_table_10 [id=60b67e8bde6c4177a72606ad2be64acf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.330427 31716 tablet_bootstrap.cc:492] T aef29abf9b4e4cc195ad65ae15b00fd5 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:38.330597 31716 ts_tablet_manager.cc:1367] T aef29abf9b4e4cc195ad65ae15b00fd5 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20240503 02:12:38.330732 29854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb7965d0d50944c6be98c27f40f46de7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.331409 31716 raft_consensus.cc:348] T aef29abf9b4e4cc195ad65ae15b00fd5 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:38.331682 31716 raft_consensus.cc:374] T aef29abf9b4e4cc195ad65ae15b00fd5 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:38.331795 31716 raft_consensus.cc:729] T aef29abf9b4e4cc195ad65ae15b00fd5 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:38.332098 31716 consensus_queue.cc:260] T aef29abf9b4e4cc195ad65ae15b00fd5 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:38.332792 31716 ts_tablet_manager.cc:1398] T aef29abf9b4e4cc195ad65ae15b00fd5 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 02:12:38.333077 31716 tablet_bootstrap.cc:492] T ad7b19f47b464333ad4c2736019c8fd6 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:38.337714 31715 raft_consensus.cc:348] T ad7b19f47b464333ad4c2736019c8fd6 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:38.338024 31715 raft_consensus.cc:374] T ad7b19f47b464333ad4c2736019c8fd6 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:38.338178 31715 raft_consensus.cc:729] T ad7b19f47b464333ad4c2736019c8fd6 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:38.338478 31715 consensus_queue.cc:260] T ad7b19f47b464333ad4c2736019c8fd6 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:38.339140 31715 ts_tablet_manager.cc:1398] T ad7b19f47b464333ad4c2736019c8fd6 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240503 02:12:38.342257 31716 tablet_bootstrap.cc:654] T ad7b19f47b464333ad4c2736019c8fd6 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:38.344071 31715 tablet_bootstrap.cc:492] T cb7965d0d50944c6be98c27f40f46de7 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:38.349541 31715 tablet_bootstrap.cc:654] T cb7965d0d50944c6be98c27f40f46de7 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:38.352883 31720 tablet_bootstrap.cc:492] T aef29abf9b4e4cc195ad65ae15b00fd5 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:38.355412 31716 tablet_bootstrap.cc:492] T ad7b19f47b464333ad4c2736019c8fd6 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:38.355424 31720 tablet_bootstrap.cc:654] T aef29abf9b4e4cc195ad65ae15b00fd5 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:38.355698 31716 ts_tablet_manager.cc:1367] T ad7b19f47b464333ad4c2736019c8fd6 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.007s
I20240503 02:12:38.356525 31716 raft_consensus.cc:348] T ad7b19f47b464333ad4c2736019c8fd6 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:38.356837 31716 raft_consensus.cc:374] T ad7b19f47b464333ad4c2736019c8fd6 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:38.356962 31716 raft_consensus.cc:729] T ad7b19f47b464333ad4c2736019c8fd6 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:38.357376 31716 consensus_queue.cc:260] T ad7b19f47b464333ad4c2736019c8fd6 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:38.358151 31716 ts_tablet_manager.cc:1398] T ad7b19f47b464333ad4c2736019c8fd6 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:38.357759 31714 tablet_bootstrap.cc:492] T ad7b19f47b464333ad4c2736019c8fd6 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:38.358848 31714 ts_tablet_manager.cc:1367] T ad7b19f47b464333ad4c2736019c8fd6 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.047s user 0.013s sys 0.009s
I20240503 02:12:38.359639 31714 raft_consensus.cc:348] T ad7b19f47b464333ad4c2736019c8fd6 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:38.359845 31714 raft_consensus.cc:374] T ad7b19f47b464333ad4c2736019c8fd6 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:38.359941 31714 raft_consensus.cc:729] T ad7b19f47b464333ad4c2736019c8fd6 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:38.360189 31714 consensus_queue.cc:260] T ad7b19f47b464333ad4c2736019c8fd6 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:38.360865 31714 ts_tablet_manager.cc:1398] T ad7b19f47b464333ad4c2736019c8fd6 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:38.361219 31714 tablet_bootstrap.cc:492] T cb7965d0d50944c6be98c27f40f46de7 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:38.364560 31715 tablet_bootstrap.cc:492] T cb7965d0d50944c6be98c27f40f46de7 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:38.364755 31715 ts_tablet_manager.cc:1367] T cb7965d0d50944c6be98c27f40f46de7 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.009s
I20240503 02:12:38.365099 31714 tablet_bootstrap.cc:654] T cb7965d0d50944c6be98c27f40f46de7 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:38.366760 31715 raft_consensus.cc:348] T cb7965d0d50944c6be98c27f40f46de7 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:38.367033 31715 raft_consensus.cc:374] T cb7965d0d50944c6be98c27f40f46de7 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:38.367113 31715 raft_consensus.cc:729] T cb7965d0d50944c6be98c27f40f46de7 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:38.367362 31715 consensus_queue.cc:260] T cb7965d0d50944c6be98c27f40f46de7 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:38.368003 31715 ts_tablet_manager.cc:1398] T cb7965d0d50944c6be98c27f40f46de7 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:12:38.370030 31715 tablet_bootstrap.cc:492] T aef29abf9b4e4cc195ad65ae15b00fd5 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:38.371665 31715 tablet_bootstrap.cc:654] T aef29abf9b4e4cc195ad65ae15b00fd5 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:38.374475 31720 tablet_bootstrap.cc:492] T aef29abf9b4e4cc195ad65ae15b00fd5 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:38.374671 31720 ts_tablet_manager.cc:1367] T aef29abf9b4e4cc195ad65ae15b00fd5 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.009s
I20240503 02:12:38.375386 31720 raft_consensus.cc:348] T aef29abf9b4e4cc195ad65ae15b00fd5 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:38.375619 31720 raft_consensus.cc:374] T aef29abf9b4e4cc195ad65ae15b00fd5 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:38.375722 31720 raft_consensus.cc:729] T aef29abf9b4e4cc195ad65ae15b00fd5 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:38.376025 31720 consensus_queue.cc:260] T aef29abf9b4e4cc195ad65ae15b00fd5 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:38.376610 31720 ts_tablet_manager.cc:1398] T aef29abf9b4e4cc195ad65ae15b00fd5 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:38.376924 31715 tablet_bootstrap.cc:492] T aef29abf9b4e4cc195ad65ae15b00fd5 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:38.377079 31715 ts_tablet_manager.cc:1367] T aef29abf9b4e4cc195ad65ae15b00fd5 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20240503 02:12:38.377825 31715 raft_consensus.cc:348] T aef29abf9b4e4cc195ad65ae15b00fd5 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:38.378026 31715 raft_consensus.cc:374] T aef29abf9b4e4cc195ad65ae15b00fd5 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:38.378122 31715 raft_consensus.cc:729] T aef29abf9b4e4cc195ad65ae15b00fd5 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:38.378316 31715 consensus_queue.cc:260] T aef29abf9b4e4cc195ad65ae15b00fd5 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:38.378949 31715 ts_tablet_manager.cc:1398] T aef29abf9b4e4cc195ad65ae15b00fd5 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:38.379494 31716 tablet_bootstrap.cc:492] T cb7965d0d50944c6be98c27f40f46de7 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:38.381453 31716 tablet_bootstrap.cc:654] T cb7965d0d50944c6be98c27f40f46de7 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:38.384014 31714 tablet_bootstrap.cc:492] T cb7965d0d50944c6be98c27f40f46de7 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:38.384181 31714 ts_tablet_manager.cc:1367] T cb7965d0d50944c6be98c27f40f46de7 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.002s
I20240503 02:12:38.384832 31714 raft_consensus.cc:348] T cb7965d0d50944c6be98c27f40f46de7 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:38.385020 31714 raft_consensus.cc:374] T cb7965d0d50944c6be98c27f40f46de7 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:38.385118 31714 raft_consensus.cc:729] T cb7965d0d50944c6be98c27f40f46de7 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:38.385370 31714 consensus_queue.cc:260] T cb7965d0d50944c6be98c27f40f46de7 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:38.385830 31714 ts_tablet_manager.cc:1398] T cb7965d0d50944c6be98c27f40f46de7 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:38.390722 31716 tablet_bootstrap.cc:492] T cb7965d0d50944c6be98c27f40f46de7 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:38.390880 31716 ts_tablet_manager.cc:1367] T cb7965d0d50944c6be98c27f40f46de7 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20240503 02:12:38.391497 31716 raft_consensus.cc:348] T cb7965d0d50944c6be98c27f40f46de7 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:38.391700 31716 raft_consensus.cc:374] T cb7965d0d50944c6be98c27f40f46de7 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:38.391819 31716 raft_consensus.cc:729] T cb7965d0d50944c6be98c27f40f46de7 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:38.392027 31716 consensus_queue.cc:260] T cb7965d0d50944c6be98c27f40f46de7 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:38.392849 31716 ts_tablet_manager.cc:1398] T cb7965d0d50944c6be98c27f40f46de7 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:38.433838 31328 raft_consensus.cc:482] T cb7965d0d50944c6be98c27f40f46de7 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:38.434032 31328 raft_consensus.cc:504] T cb7965d0d50944c6be98c27f40f46de7 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:38.434756 31328 leader_election.cc:288] T cb7965d0d50944c6be98c27f40f46de7 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:12:38.435182 29874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb7965d0d50944c6be98c27f40f46de7" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:38.435456 29874 raft_consensus.cc:2454] T cb7965d0d50944c6be98c27f40f46de7 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:12:38.435516 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb7965d0d50944c6be98c27f40f46de7" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:12:38.435758 30264 raft_consensus.cc:2454] T cb7965d0d50944c6be98c27f40f46de7 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:12:38.435956 29942 leader_election.cc:302] T cb7965d0d50944c6be98c27f40f46de7 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429; no voters:
I20240503 02:12:38.436271 31328 raft_consensus.cc:2781] T cb7965d0d50944c6be98c27f40f46de7 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:38.436389 31328 raft_consensus.cc:482] T cb7965d0d50944c6be98c27f40f46de7 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:38.436491 31328 raft_consensus.cc:3037] T cb7965d0d50944c6be98c27f40f46de7 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:38.439293 31328 raft_consensus.cc:504] T cb7965d0d50944c6be98c27f40f46de7 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:38.439962 31328 leader_election.cc:288] T cb7965d0d50944c6be98c27f40f46de7 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Requested vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:12:38.440393 29874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb7965d0d50944c6be98c27f40f46de7" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:12:38.440450 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb7965d0d50944c6be98c27f40f46de7" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:12:38.440589 29874 raft_consensus.cc:3037] T cb7965d0d50944c6be98c27f40f46de7 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:38.440637 30264 raft_consensus.cc:3037] T cb7965d0d50944c6be98c27f40f46de7 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:38.442996 30264 raft_consensus.cc:2454] T cb7965d0d50944c6be98c27f40f46de7 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:12:38.443020 29874 raft_consensus.cc:2454] T cb7965d0d50944c6be98c27f40f46de7 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:12:38.443457 29942 leader_election.cc:302] T cb7965d0d50944c6be98c27f40f46de7 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:38.443737 31328 raft_consensus.cc:2781] T cb7965d0d50944c6be98c27f40f46de7 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:38.444105 31328 raft_consensus.cc:686] T cb7965d0d50944c6be98c27f40f46de7 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Becoming Leader. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Running, Role: LEADER
I20240503 02:12:38.444379 31328 consensus_queue.cc:237] T cb7965d0d50944c6be98c27f40f46de7 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:38.447494 29731 catalog_manager.cc:5484] T cb7965d0d50944c6be98c27f40f46de7 P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: term changed from 0 to 1, leader changed from <none> to 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2). New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:12:38.490082 31604 raft_consensus.cc:482] T aef29abf9b4e4cc195ad65ae15b00fd5 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:38.490285 31604 raft_consensus.cc:504] T aef29abf9b4e4cc195ad65ae15b00fd5 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:38.490954 31604 leader_election.cc:288] T aef29abf9b4e4cc195ad65ae15b00fd5 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:12:38.491504 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aef29abf9b4e4cc195ad65ae15b00fd5" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:12:38.491580 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aef29abf9b4e4cc195ad65ae15b00fd5" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:12:38.491704 30005 raft_consensus.cc:2454] T aef29abf9b4e4cc195ad65ae15b00fd5 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 0.
I20240503 02:12:38.491830 30264 raft_consensus.cc:2454] T aef29abf9b4e4cc195ad65ae15b00fd5 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 0.
I20240503 02:12:38.492182 30070 leader_election.cc:302] T aef29abf9b4e4cc195ad65ae15b00fd5 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:38.492450 31604 raft_consensus.cc:2781] T aef29abf9b4e4cc195ad65ae15b00fd5 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:38.492558 31604 raft_consensus.cc:482] T aef29abf9b4e4cc195ad65ae15b00fd5 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:38.492668 31604 raft_consensus.cc:3037] T aef29abf9b4e4cc195ad65ae15b00fd5 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:38.494611 31604 raft_consensus.cc:504] T aef29abf9b4e4cc195ad65ae15b00fd5 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:38.495123 31604 leader_election.cc:288] T aef29abf9b4e4cc195ad65ae15b00fd5 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Requested vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:12:38.495653 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aef29abf9b4e4cc195ad65ae15b00fd5" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:12:38.495682 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aef29abf9b4e4cc195ad65ae15b00fd5" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:12:38.495867 30005 raft_consensus.cc:3037] T aef29abf9b4e4cc195ad65ae15b00fd5 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:38.495865 30264 raft_consensus.cc:3037] T aef29abf9b4e4cc195ad65ae15b00fd5 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:38.497913 30264 raft_consensus.cc:2454] T aef29abf9b4e4cc195ad65ae15b00fd5 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 1.
I20240503 02:12:38.497978 30005 raft_consensus.cc:2454] T aef29abf9b4e4cc195ad65ae15b00fd5 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 1.
I20240503 02:12:38.498292 30070 leader_election.cc:302] T aef29abf9b4e4cc195ad65ae15b00fd5 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:38.498575 31604 raft_consensus.cc:2781] T aef29abf9b4e4cc195ad65ae15b00fd5 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:38.498720 31604 raft_consensus.cc:686] T aef29abf9b4e4cc195ad65ae15b00fd5 P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Becoming Leader. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Running, Role: LEADER
I20240503 02:12:38.498955 31604 consensus_queue.cc:237] T aef29abf9b4e4cc195ad65ae15b00fd5 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:38.501798 29731 catalog_manager.cc:5484] T aef29abf9b4e4cc195ad65ae15b00fd5 P a9094240860e43bfb129193f3bfd5be6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9094240860e43bfb129193f3bfd5be6 (127.22.108.3). New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:12:38.531184 31709 raft_consensus.cc:482] T ad7b19f47b464333ad4c2736019c8fd6 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:38.531358 31709 raft_consensus.cc:504] T ad7b19f47b464333ad4c2736019c8fd6 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:38.532065 31709 leader_election.cc:288] T ad7b19f47b464333ad4c2736019c8fd6 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:38.532462 29874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7b19f47b464333ad4c2736019c8fd6" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:38.532462 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7b19f47b464333ad4c2736019c8fd6" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:12:38.532722 29874 raft_consensus.cc:2454] T ad7b19f47b464333ad4c2736019c8fd6 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:38.532717 30005 raft_consensus.cc:2454] T ad7b19f47b464333ad4c2736019c8fd6 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:38.533134 30200 leader_election.cc:302] T ad7b19f47b464333ad4c2736019c8fd6 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:38.533535 31730 raft_consensus.cc:2781] T ad7b19f47b464333ad4c2736019c8fd6 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:38.533636 31730 raft_consensus.cc:482] T ad7b19f47b464333ad4c2736019c8fd6 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:38.533762 31730 raft_consensus.cc:3037] T ad7b19f47b464333ad4c2736019c8fd6 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:38.535557 31730 raft_consensus.cc:504] T ad7b19f47b464333ad4c2736019c8fd6 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:38.536100 31730 leader_election.cc:288] T ad7b19f47b464333ad4c2736019c8fd6 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Requested vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:38.536466 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7b19f47b464333ad4c2736019c8fd6" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:12:38.536545 29874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7b19f47b464333ad4c2736019c8fd6" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:12:38.536660 30005 raft_consensus.cc:3037] T ad7b19f47b464333ad4c2736019c8fd6 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:38.536818 29874 raft_consensus.cc:3037] T ad7b19f47b464333ad4c2736019c8fd6 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:38.538520 30005 raft_consensus.cc:2454] T ad7b19f47b464333ad4c2736019c8fd6 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:38.538741 29874 raft_consensus.cc:2454] T ad7b19f47b464333ad4c2736019c8fd6 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:38.538975 30200 leader_election.cc:302] T ad7b19f47b464333ad4c2736019c8fd6 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:38.539242 31730 raft_consensus.cc:2781] T ad7b19f47b464333ad4c2736019c8fd6 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:38.539397 31730 raft_consensus.cc:686] T ad7b19f47b464333ad4c2736019c8fd6 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Becoming Leader. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Running, Role: LEADER
I20240503 02:12:38.539878 31730 consensus_queue.cc:237] T ad7b19f47b464333ad4c2736019c8fd6 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:38.542681 29731 catalog_manager.cc:5484] T ad7b19f47b464333ad4c2736019c8fd6 P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4). New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:12:38.939311 31328 consensus_queue.cc:1035] T cb7965d0d50944c6be98c27f40f46de7 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:38.946066 31328 consensus_queue.cc:1035] T cb7965d0d50944c6be98c27f40f46de7 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:38.947232 31732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:38.947685 31732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:38.957662 31732 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240503 02:12:38.996274 31604 consensus_queue.cc:1035] T aef29abf9b4e4cc195ad65ae15b00fd5 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:39.002058 31604 consensus_queue.cc:1035] T aef29abf9b4e4cc195ad65ae15b00fd5 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:39.024760 31730 consensus_queue.cc:1035] T ad7b19f47b464333ad4c2736019c8fd6 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:39.032536 31730 consensus_queue.cc:1035] T ad7b19f47b464333ad4c2736019c8fd6 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 233.699 ms
time per row: 0.11685 ms
I20240503 02:12:39.348163 29732 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:39.365500 30116 tablet_service.cc:1467] Processing CreateTablet for tablet 5f4b72b138c64ac58e6477cd2b1fb215 (DEFAULT_TABLE table=rebalance_test_table_11 [id=9498d0be60bf4564a9127c17a357f7ed]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.365707 29853 tablet_service.cc:1467] Processing CreateTablet for tablet 5f4b72b138c64ac58e6477cd2b1fb215 (DEFAULT_TABLE table=rebalance_test_table_11 [id=9498d0be60bf4564a9127c17a357f7ed]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.366037 30116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f4b72b138c64ac58e6477cd2b1fb215. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.366160 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f4b72b138c64ac58e6477cd2b1fb215. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.367344 29984 tablet_service.cc:1467] Processing CreateTablet for tablet 04e47da8cf7445dcacaf1f7c8950bd34 (DEFAULT_TABLE table=rebalance_test_table_11 [id=9498d0be60bf4564a9127c17a357f7ed]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.367771 29984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04e47da8cf7445dcacaf1f7c8950bd34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.367933 30114 tablet_service.cc:1467] Processing CreateTablet for tablet c19017f829a74cc2b6dc41a2a25a3831 (DEFAULT_TABLE table=rebalance_test_table_11 [id=9498d0be60bf4564a9127c17a357f7ed]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.368325 30114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c19017f829a74cc2b6dc41a2a25a3831. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.372721 29985 tablet_service.cc:1467] Processing CreateTablet for tablet 5f4b72b138c64ac58e6477cd2b1fb215 (DEFAULT_TABLE table=rebalance_test_table_11 [id=9498d0be60bf4564a9127c17a357f7ed]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.373119 29985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f4b72b138c64ac58e6477cd2b1fb215. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.373467 30115 tablet_service.cc:1467] Processing CreateTablet for tablet 04e47da8cf7445dcacaf1f7c8950bd34 (DEFAULT_TABLE table=rebalance_test_table_11 [id=9498d0be60bf4564a9127c17a357f7ed]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.373982 30115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04e47da8cf7445dcacaf1f7c8950bd34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.374111 30246 tablet_service.cc:1467] Processing CreateTablet for tablet 04e47da8cf7445dcacaf1f7c8950bd34 (DEFAULT_TABLE table=rebalance_test_table_11 [id=9498d0be60bf4564a9127c17a357f7ed]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.374523 30246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04e47da8cf7445dcacaf1f7c8950bd34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.377440 30244 tablet_service.cc:1467] Processing CreateTablet for tablet c19017f829a74cc2b6dc41a2a25a3831 (DEFAULT_TABLE table=rebalance_test_table_11 [id=9498d0be60bf4564a9127c17a357f7ed]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.377663 31800 tablet_bootstrap.cc:492] T 5f4b72b138c64ac58e6477cd2b1fb215 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:39.377818 30244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c19017f829a74cc2b6dc41a2a25a3831. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.380004 31800 tablet_bootstrap.cc:654] T 5f4b72b138c64ac58e6477cd2b1fb215 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.381482 29854 tablet_service.cc:1467] Processing CreateTablet for tablet c19017f829a74cc2b6dc41a2a25a3831 (DEFAULT_TABLE table=rebalance_test_table_11 [id=9498d0be60bf4564a9127c17a357f7ed]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.381923 29854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c19017f829a74cc2b6dc41a2a25a3831. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.384299 31802 tablet_bootstrap.cc:492] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:39.386837 31802 tablet_bootstrap.cc:654] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.392261 31803 tablet_bootstrap.cc:492] T 04e47da8cf7445dcacaf1f7c8950bd34 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:39.393596 31801 tablet_bootstrap.cc:492] T 5f4b72b138c64ac58e6477cd2b1fb215 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:39.394243 31803 tablet_bootstrap.cc:654] T 04e47da8cf7445dcacaf1f7c8950bd34 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.395704 31802 tablet_bootstrap.cc:492] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:39.395864 31802 ts_tablet_manager.cc:1367] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240503 02:12:39.396498 31801 tablet_bootstrap.cc:654] T 5f4b72b138c64ac58e6477cd2b1fb215 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.396783 31803 tablet_bootstrap.cc:492] T 04e47da8cf7445dcacaf1f7c8950bd34 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:39.396711 31802 raft_consensus.cc:348] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:39.396942 31802 raft_consensus.cc:374] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.397059 31802 raft_consensus.cc:729] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.397264 31803 ts_tablet_manager.cc:1367] T 04e47da8cf7445dcacaf1f7c8950bd34 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240503 02:12:39.397303 31802 consensus_queue.cc:260] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:39.399072 31802 ts_tablet_manager.cc:1398] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:39.399333 31802 tablet_bootstrap.cc:492] T 5f4b72b138c64ac58e6477cd2b1fb215 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:39.399540 31803 raft_consensus.cc:348] T 04e47da8cf7445dcacaf1f7c8950bd34 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:39.399845 31803 raft_consensus.cc:374] T 04e47da8cf7445dcacaf1f7c8950bd34 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.399973 31803 raft_consensus.cc:729] T 04e47da8cf7445dcacaf1f7c8950bd34 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.400271 31803 consensus_queue.cc:260] T 04e47da8cf7445dcacaf1f7c8950bd34 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:39.400857 31802 tablet_bootstrap.cc:654] T 5f4b72b138c64ac58e6477cd2b1fb215 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.401736 31803 ts_tablet_manager.cc:1398] T 04e47da8cf7445dcacaf1f7c8950bd34 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:12:39.402076 31803 tablet_bootstrap.cc:492] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:39.402953 31800 tablet_bootstrap.cc:492] T 5f4b72b138c64ac58e6477cd2b1fb215 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:39.403131 31800 ts_tablet_manager.cc:1367] T 5f4b72b138c64ac58e6477cd2b1fb215 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.026s user 0.010s sys 0.005s
I20240503 02:12:39.403750 31803 tablet_bootstrap.cc:654] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.406999 31803 tablet_bootstrap.cc:492] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:39.406867 31800 raft_consensus.cc:348] T 5f4b72b138c64ac58e6477cd2b1fb215 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:39.407091 31800 raft_consensus.cc:374] T 5f4b72b138c64ac58e6477cd2b1fb215 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.407193 31800 raft_consensus.cc:729] T 5f4b72b138c64ac58e6477cd2b1fb215 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.407248 31803 ts_tablet_manager.cc:1367] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20240503 02:12:39.407460 31800 consensus_queue.cc:260] T 5f4b72b138c64ac58e6477cd2b1fb215 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:39.408176 31800 ts_tablet_manager.cc:1398] T 5f4b72b138c64ac58e6477cd2b1fb215 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 02:12:39.408123 31803 raft_consensus.cc:348] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:39.408339 31803 raft_consensus.cc:374] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.408409 31803 raft_consensus.cc:729] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.408483 31800 tablet_bootstrap.cc:492] T 04e47da8cf7445dcacaf1f7c8950bd34 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:39.408644 31803 consensus_queue.cc:260] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:39.410169 31800 tablet_bootstrap.cc:654] T 04e47da8cf7445dcacaf1f7c8950bd34 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.410502 31803 ts_tablet_manager.cc:1398] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:12:39.413242 31801 tablet_bootstrap.cc:492] T 5f4b72b138c64ac58e6477cd2b1fb215 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:39.414410 31802 tablet_bootstrap.cc:492] T 5f4b72b138c64ac58e6477cd2b1fb215 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:39.414593 31802 ts_tablet_manager.cc:1367] T 5f4b72b138c64ac58e6477cd2b1fb215 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20240503 02:12:39.414542 31801 ts_tablet_manager.cc:1367] T 5f4b72b138c64ac58e6477cd2b1fb215 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.009s
I20240503 02:12:39.415452 31801 raft_consensus.cc:348] T 5f4b72b138c64ac58e6477cd2b1fb215 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:39.415472 31802 raft_consensus.cc:348] T 5f4b72b138c64ac58e6477cd2b1fb215 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:39.415710 31802 raft_consensus.cc:374] T 5f4b72b138c64ac58e6477cd2b1fb215 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.415796 31802 raft_consensus.cc:729] T 5f4b72b138c64ac58e6477cd2b1fb215 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.415802 31801 raft_consensus.cc:374] T 5f4b72b138c64ac58e6477cd2b1fb215 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.415968 31801 raft_consensus.cc:729] T 5f4b72b138c64ac58e6477cd2b1fb215 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.416072 31802 consensus_queue.cc:260] T 5f4b72b138c64ac58e6477cd2b1fb215 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:39.416270 31801 consensus_queue.cc:260] T 5f4b72b138c64ac58e6477cd2b1fb215 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:39.416632 31802 ts_tablet_manager.cc:1398] T 5f4b72b138c64ac58e6477cd2b1fb215 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:39.417399 31801 ts_tablet_manager.cc:1398] T 5f4b72b138c64ac58e6477cd2b1fb215 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:39.417904 31801 tablet_bootstrap.cc:492] T c19017f829a74cc2b6dc41a2a25a3831 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:39.419140 31800 tablet_bootstrap.cc:492] T 04e47da8cf7445dcacaf1f7c8950bd34 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:39.419329 31800 ts_tablet_manager.cc:1367] T 04e47da8cf7445dcacaf1f7c8950bd34 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240503 02:12:39.419608 31801 tablet_bootstrap.cc:654] T c19017f829a74cc2b6dc41a2a25a3831 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.420171 31800 raft_consensus.cc:348] T 04e47da8cf7445dcacaf1f7c8950bd34 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:39.420416 31800 raft_consensus.cc:374] T 04e47da8cf7445dcacaf1f7c8950bd34 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.420502 31800 raft_consensus.cc:729] T 04e47da8cf7445dcacaf1f7c8950bd34 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.420837 31800 consensus_queue.cc:260] T 04e47da8cf7445dcacaf1f7c8950bd34 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:39.421422 31800 ts_tablet_manager.cc:1398] T 04e47da8cf7445dcacaf1f7c8950bd34 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:39.422073 31801 tablet_bootstrap.cc:492] T c19017f829a74cc2b6dc41a2a25a3831 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:39.422243 31801 ts_tablet_manager.cc:1367] T c19017f829a74cc2b6dc41a2a25a3831 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:12:39.422909 31800 tablet_bootstrap.cc:492] T c19017f829a74cc2b6dc41a2a25a3831 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:39.422951 31801 raft_consensus.cc:348] T c19017f829a74cc2b6dc41a2a25a3831 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:39.424186 31801 raft_consensus.cc:374] T c19017f829a74cc2b6dc41a2a25a3831 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.424237 31800 tablet_bootstrap.cc:654] T c19017f829a74cc2b6dc41a2a25a3831 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.424348 31801 raft_consensus.cc:729] T c19017f829a74cc2b6dc41a2a25a3831 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.424687 31801 consensus_queue.cc:260] T c19017f829a74cc2b6dc41a2a25a3831 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:39.425318 31801 ts_tablet_manager.cc:1398] T c19017f829a74cc2b6dc41a2a25a3831 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:39.438225 31800 tablet_bootstrap.cc:492] T c19017f829a74cc2b6dc41a2a25a3831 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:39.438369 31800 ts_tablet_manager.cc:1367] T c19017f829a74cc2b6dc41a2a25a3831 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.009s
I20240503 02:12:39.439062 31800 raft_consensus.cc:348] T c19017f829a74cc2b6dc41a2a25a3831 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:39.439257 31800 raft_consensus.cc:374] T c19017f829a74cc2b6dc41a2a25a3831 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.439352 31800 raft_consensus.cc:729] T c19017f829a74cc2b6dc41a2a25a3831 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.439612 31800 consensus_queue.cc:260] T c19017f829a74cc2b6dc41a2a25a3831 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:39.440174 31800 ts_tablet_manager.cc:1398] T c19017f829a74cc2b6dc41a2a25a3831 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:39.478492 31790 raft_consensus.cc:482] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:39.478682 31790 raft_consensus.cc:504] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:39.479463 31790 leader_election.cc:288] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:12:39.479928 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e47da8cf7445dcacaf1f7c8950bd34" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:12:39.480082 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e47da8cf7445dcacaf1f7c8950bd34" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:12:39.480178 30134 raft_consensus.cc:2454] T 04e47da8cf7445dcacaf1f7c8950bd34 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:12:39.480334 30266 raft_consensus.cc:2454] T 04e47da8cf7445dcacaf1f7c8950bd34 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:12:39.480713 29939 leader_election.cc:302] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:39.480990 31790 raft_consensus.cc:2781] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:39.481138 31790 raft_consensus.cc:482] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:39.481238 31790 raft_consensus.cc:3037] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:39.483184 31790 raft_consensus.cc:504] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:39.483747 31790 leader_election.cc:288] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Requested vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:12:39.484223 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e47da8cf7445dcacaf1f7c8950bd34" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:12:39.484230 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e47da8cf7445dcacaf1f7c8950bd34" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:12:39.484423 30266 raft_consensus.cc:3037] T 04e47da8cf7445dcacaf1f7c8950bd34 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:39.484421 30134 raft_consensus.cc:3037] T 04e47da8cf7445dcacaf1f7c8950bd34 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:39.486810 30266 raft_consensus.cc:2454] T 04e47da8cf7445dcacaf1f7c8950bd34 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:12:39.486809 30134 raft_consensus.cc:2454] T 04e47da8cf7445dcacaf1f7c8950bd34 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:12:39.487200 29940 leader_election.cc:302] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:39.487682 31790 raft_consensus.cc:2781] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:39.487844 31790 raft_consensus.cc:686] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Becoming Leader. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Running, Role: LEADER
I20240503 02:12:39.488148 31790 consensus_queue.cc:237] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:39.491423 29732 catalog_manager.cc:5484] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: term changed from 0 to 1, leader changed from <none> to 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2). New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:12:39.496240 31765 raft_consensus.cc:482] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:39.496388 31765 raft_consensus.cc:504] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:39.496986 31765 leader_election.cc:288] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:39.497426 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c19017f829a74cc2b6dc41a2a25a3831" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:12:39.497475 29874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c19017f829a74cc2b6dc41a2a25a3831" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:39.497646 30134 raft_consensus.cc:2454] T c19017f829a74cc2b6dc41a2a25a3831 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:39.497687 29874 raft_consensus.cc:2454] T c19017f829a74cc2b6dc41a2a25a3831 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:39.498061 30202 leader_election.cc:302] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:39.498312 31765 raft_consensus.cc:2781] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:39.498437 31765 raft_consensus.cc:482] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:39.498540 31765 raft_consensus.cc:3037] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:39.500308 31765 raft_consensus.cc:504] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:39.500768 31765 leader_election.cc:288] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Requested vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:39.501188 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c19017f829a74cc2b6dc41a2a25a3831" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:12:39.501339 30134 raft_consensus.cc:3037] T c19017f829a74cc2b6dc41a2a25a3831 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:39.501325 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c19017f829a74cc2b6dc41a2a25a3831" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:12:39.501535 29875 raft_consensus.cc:3037] T c19017f829a74cc2b6dc41a2a25a3831 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:39.503190 30134 raft_consensus.cc:2454] T c19017f829a74cc2b6dc41a2a25a3831 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:39.503322 29875 raft_consensus.cc:2454] T c19017f829a74cc2b6dc41a2a25a3831 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:39.503576 30199 leader_election.cc:302] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:39.503860 31765 raft_consensus.cc:2781] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:39.504001 31765 raft_consensus.cc:686] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Becoming Leader. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Running, Role: LEADER
I20240503 02:12:39.504278 31765 consensus_queue.cc:237] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:39.507304 29732 catalog_manager.cc:5484] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4). New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:12:39.537122 31733 raft_consensus.cc:482] T 5f4b72b138c64ac58e6477cd2b1fb215 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:39.537314 31733 raft_consensus.cc:504] T 5f4b72b138c64ac58e6477cd2b1fb215 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:39.537972 31733 leader_election.cc:288] T 5f4b72b138c64ac58e6477cd2b1fb215 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:12:39.538532 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4b72b138c64ac58e6477cd2b1fb215" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:12:39.538558 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4b72b138c64ac58e6477cd2b1fb215" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:12:39.538784 30004 raft_consensus.cc:2454] T 5f4b72b138c64ac58e6477cd2b1fb215 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:12:39.538807 30134 raft_consensus.cc:2454] T 5f4b72b138c64ac58e6477cd2b1fb215 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:12:39.539232 29808 leader_election.cc:302] T 5f4b72b138c64ac58e6477cd2b1fb215 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:39.539513 31733 raft_consensus.cc:2781] T 5f4b72b138c64ac58e6477cd2b1fb215 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:39.539619 31733 raft_consensus.cc:482] T 5f4b72b138c64ac58e6477cd2b1fb215 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:39.539712 31733 raft_consensus.cc:3037] T 5f4b72b138c64ac58e6477cd2b1fb215 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:39.541796 31733 raft_consensus.cc:504] T 5f4b72b138c64ac58e6477cd2b1fb215 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:39.542330 31733 leader_election.cc:288] T 5f4b72b138c64ac58e6477cd2b1fb215 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Requested vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:12:39.542748 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4b72b138c64ac58e6477cd2b1fb215" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:12:39.542831 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4b72b138c64ac58e6477cd2b1fb215" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:12:39.542966 30134 raft_consensus.cc:3037] T 5f4b72b138c64ac58e6477cd2b1fb215 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:39.543043 30004 raft_consensus.cc:3037] T 5f4b72b138c64ac58e6477cd2b1fb215 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:39.544883 30134 raft_consensus.cc:2454] T 5f4b72b138c64ac58e6477cd2b1fb215 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:12:39.545174 30004 raft_consensus.cc:2454] T 5f4b72b138c64ac58e6477cd2b1fb215 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:12:39.545313 29808 leader_election.cc:302] T 5f4b72b138c64ac58e6477cd2b1fb215 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:39.545606 31733 raft_consensus.cc:2781] T 5f4b72b138c64ac58e6477cd2b1fb215 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:39.546015 31733 raft_consensus.cc:686] T 5f4b72b138c64ac58e6477cd2b1fb215 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Becoming Leader. State: Replica: 16440e365430405fb10366ab70e2b429, State: Running, Role: LEADER
I20240503 02:12:39.546290 31733 consensus_queue.cc:237] T 5f4b72b138c64ac58e6477cd2b1fb215 P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:39.549016 29732 catalog_manager.cc:5484] T 5f4b72b138c64ac58e6477cd2b1fb215 P 16440e365430405fb10366ab70e2b429 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16440e365430405fb10366ab70e2b429 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:12:39.902576 31709 consensus_queue.cc:1035] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:39.910619 31765 consensus_queue.cc:1035] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:39.922650 31790 consensus_queue.cc:1035] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:39.929546 31328 consensus_queue.cc:1035] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:39.947571 31817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:39.947927 31817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:39.955495 31817 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:12:39.985069 31733 consensus_queue.cc:1035] T 5f4b72b138c64ac58e6477cd2b1fb215 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_11'
I20240503 02:12:39.991281 31815 consensus_queue.cc:1035] T 5f4b72b138c64ac58e6477cd2b1fb215 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 242.138 ms
time per row: 0.121069 ms
I20240503 02:12:40.350288 29729 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalance_test_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:40.359436 29985 tablet_service.cc:1467] Processing CreateTablet for tablet a592cdf6d1d747c68b5334b90006fa8b (DEFAULT_TABLE table=rebalance_test_table_12 [id=c501c7cdd0d142858bc57dffeae345f8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.359872 29985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a592cdf6d1d747c68b5334b90006fa8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.360978 29855 tablet_service.cc:1467] Processing CreateTablet for tablet a74d38f7d6e5441396944c52acd41c74 (DEFAULT_TABLE table=rebalance_test_table_12 [id=c501c7cdd0d142858bc57dffeae345f8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.361009 29853 tablet_service.cc:1467] Processing CreateTablet for tablet a592cdf6d1d747c68b5334b90006fa8b (DEFAULT_TABLE table=rebalance_test_table_12 [id=c501c7cdd0d142858bc57dffeae345f8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.361465 29855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a74d38f7d6e5441396944c52acd41c74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.361469 30115 tablet_service.cc:1467] Processing CreateTablet for tablet a74d38f7d6e5441396944c52acd41c74 (DEFAULT_TABLE table=rebalance_test_table_12 [id=c501c7cdd0d142858bc57dffeae345f8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.361702 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a592cdf6d1d747c68b5334b90006fa8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.361866 30115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a74d38f7d6e5441396944c52acd41c74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.362154 30246 tablet_service.cc:1467] Processing CreateTablet for tablet 445caf2423c1488b8e47474032e68488 (DEFAULT_TABLE table=rebalance_test_table_12 [id=c501c7cdd0d142858bc57dffeae345f8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.362533 30246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 445caf2423c1488b8e47474032e68488. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.364648 30244 tablet_service.cc:1467] Processing CreateTablet for tablet a74d38f7d6e5441396944c52acd41c74 (DEFAULT_TABLE table=rebalance_test_table_12 [id=c501c7cdd0d142858bc57dffeae345f8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.365056 30244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a74d38f7d6e5441396944c52acd41c74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.365417 29854 tablet_service.cc:1467] Processing CreateTablet for tablet 445caf2423c1488b8e47474032e68488 (DEFAULT_TABLE table=rebalance_test_table_12 [id=c501c7cdd0d142858bc57dffeae345f8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.365804 29854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 445caf2423c1488b8e47474032e68488. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.365805 29984 tablet_service.cc:1467] Processing CreateTablet for tablet 445caf2423c1488b8e47474032e68488 (DEFAULT_TABLE table=rebalance_test_table_12 [id=c501c7cdd0d142858bc57dffeae345f8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.366153 29984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 445caf2423c1488b8e47474032e68488. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.372296 30114 tablet_service.cc:1467] Processing CreateTablet for tablet a592cdf6d1d747c68b5334b90006fa8b (DEFAULT_TABLE table=rebalance_test_table_12 [id=c501c7cdd0d142858bc57dffeae345f8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.372828 30114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a592cdf6d1d747c68b5334b90006fa8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.376758 31887 tablet_bootstrap.cc:492] T a592cdf6d1d747c68b5334b90006fa8b P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:40.383170 31887 tablet_bootstrap.cc:654] T a592cdf6d1d747c68b5334b90006fa8b P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.383489 31886 tablet_bootstrap.cc:492] T a592cdf6d1d747c68b5334b90006fa8b P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:40.388983 31886 tablet_bootstrap.cc:654] T a592cdf6d1d747c68b5334b90006fa8b P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.389537 31889 tablet_bootstrap.cc:492] T a592cdf6d1d747c68b5334b90006fa8b P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:40.390436 31888 tablet_bootstrap.cc:492] T 445caf2423c1488b8e47474032e68488 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:40.391423 31887 tablet_bootstrap.cc:492] T a592cdf6d1d747c68b5334b90006fa8b P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:40.391624 31887 ts_tablet_manager.cc:1367] T a592cdf6d1d747c68b5334b90006fa8b P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240503 02:12:40.392561 31888 tablet_bootstrap.cc:654] T 445caf2423c1488b8e47474032e68488 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.392541 31887 raft_consensus.cc:348] T a592cdf6d1d747c68b5334b90006fa8b P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:40.393002 31889 tablet_bootstrap.cc:654] T a592cdf6d1d747c68b5334b90006fa8b P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.393232 31886 tablet_bootstrap.cc:492] T a592cdf6d1d747c68b5334b90006fa8b P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:40.393445 31886 ts_tablet_manager.cc:1367] T a592cdf6d1d747c68b5334b90006fa8b P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.001s
I20240503 02:12:40.393741 31887 raft_consensus.cc:374] T a592cdf6d1d747c68b5334b90006fa8b P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.393935 31887 raft_consensus.cc:729] T a592cdf6d1d747c68b5334b90006fa8b P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.394196 31887 consensus_queue.cc:260] T a592cdf6d1d747c68b5334b90006fa8b P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:40.394793 31887 ts_tablet_manager.cc:1398] T a592cdf6d1d747c68b5334b90006fa8b P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:40.394627 31886 raft_consensus.cc:348] T a592cdf6d1d747c68b5334b90006fa8b P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:40.395148 31886 raft_consensus.cc:374] T a592cdf6d1d747c68b5334b90006fa8b P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.395213 31887 tablet_bootstrap.cc:492] T 445caf2423c1488b8e47474032e68488 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:40.396692 31887 tablet_bootstrap.cc:654] T 445caf2423c1488b8e47474032e68488 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.398447 31826 raft_consensus.cc:482] T a592cdf6d1d747c68b5334b90006fa8b P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:40.398626 31826 raft_consensus.cc:504] T a592cdf6d1d747c68b5334b90006fa8b P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:40.395335 31886 raft_consensus.cc:729] T a592cdf6d1d747c68b5334b90006fa8b P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.399637 31886 consensus_queue.cc:260] T a592cdf6d1d747c68b5334b90006fa8b P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:40.399803 31826 leader_election.cc:288] T a592cdf6d1d747c68b5334b90006fa8b P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:40.400666 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a592cdf6d1d747c68b5334b90006fa8b" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:40.400960 29875 raft_consensus.cc:2454] T a592cdf6d1d747c68b5334b90006fa8b P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:12:40.403537 29942 leader_election.cc:302] T a592cdf6d1d747c68b5334b90006fa8b P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429; no voters:
I20240503 02:12:40.403529 30136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a592cdf6d1d747c68b5334b90006fa8b" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:12:40.403829 31826 raft_consensus.cc:2781] T a592cdf6d1d747c68b5334b90006fa8b P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:40.403967 31826 raft_consensus.cc:482] T a592cdf6d1d747c68b5334b90006fa8b P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:40.404076 31826 raft_consensus.cc:3037] T a592cdf6d1d747c68b5334b90006fa8b P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:40.410072 29939 leader_election.cc:341] T a592cdf6d1d747c68b5334b90006fa8b P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:40.415457 31886 ts_tablet_manager.cc:1398] T a592cdf6d1d747c68b5334b90006fa8b P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.022s user 0.001s sys 0.002s
I20240503 02:12:40.416020 31886 tablet_bootstrap.cc:492] T a74d38f7d6e5441396944c52acd41c74 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:40.420238 31886 tablet_bootstrap.cc:654] T a74d38f7d6e5441396944c52acd41c74 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.408663 31826 raft_consensus.cc:504] T a592cdf6d1d747c68b5334b90006fa8b P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:40.422144 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a592cdf6d1d747c68b5334b90006fa8b" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:12:40.422420 29875 raft_consensus.cc:3037] T a592cdf6d1d747c68b5334b90006fa8b P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:40.426482 29875 raft_consensus.cc:2454] T a592cdf6d1d747c68b5334b90006fa8b P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:12:40.427050 31826 leader_election.cc:288] T a592cdf6d1d747c68b5334b90006fa8b P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Requested vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:40.427982 30136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a592cdf6d1d747c68b5334b90006fa8b" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
W20240503 02:12:40.428735 29939 leader_election.cc:341] T a592cdf6d1d747c68b5334b90006fa8b P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:40.431463 29942 leader_election.cc:302] T a592cdf6d1d747c68b5334b90006fa8b P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429; no voters: a9094240860e43bfb129193f3bfd5be6
I20240503 02:12:40.432278 31328 raft_consensus.cc:2781] T a592cdf6d1d747c68b5334b90006fa8b P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:40.432451 31328 raft_consensus.cc:686] T a592cdf6d1d747c68b5334b90006fa8b P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Becoming Leader. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Running, Role: LEADER
I20240503 02:12:40.432808 31328 consensus_queue.cc:237] T a592cdf6d1d747c68b5334b90006fa8b P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:40.440197 29729 catalog_manager.cc:5484] T a592cdf6d1d747c68b5334b90006fa8b P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: term changed from 0 to 1, leader changed from <none> to 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2). New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:12:40.449859 31886 tablet_bootstrap.cc:492] T a74d38f7d6e5441396944c52acd41c74 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:40.450228 31886 ts_tablet_manager.cc:1367] T a74d38f7d6e5441396944c52acd41c74 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.002s
I20240503 02:12:40.451186 31889 tablet_bootstrap.cc:492] T a592cdf6d1d747c68b5334b90006fa8b P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:40.451459 31889 ts_tablet_manager.cc:1367] T a592cdf6d1d747c68b5334b90006fa8b P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.062s user 0.001s sys 0.005s
I20240503 02:12:40.451864 31886 raft_consensus.cc:348] T a74d38f7d6e5441396944c52acd41c74 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:40.452605 31886 raft_consensus.cc:374] T a74d38f7d6e5441396944c52acd41c74 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.452518 31889 raft_consensus.cc:348] T a592cdf6d1d747c68b5334b90006fa8b P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:40.452755 31886 raft_consensus.cc:729] T a74d38f7d6e5441396944c52acd41c74 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.452912 31889 raft_consensus.cc:374] T a592cdf6d1d747c68b5334b90006fa8b P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.453161 31889 raft_consensus.cc:729] T a592cdf6d1d747c68b5334b90006fa8b P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.453158 31886 consensus_queue.cc:260] T a74d38f7d6e5441396944c52acd41c74 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:40.453579 31889 consensus_queue.cc:260] T a592cdf6d1d747c68b5334b90006fa8b P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:40.454380 31889 ts_tablet_manager.cc:1398] T a592cdf6d1d747c68b5334b90006fa8b P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:40.457619 31886 ts_tablet_manager.cc:1398] T a74d38f7d6e5441396944c52acd41c74 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:12:40.458007 31886 tablet_bootstrap.cc:492] T 445caf2423c1488b8e47474032e68488 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:40.461447 31889 tablet_bootstrap.cc:492] T a74d38f7d6e5441396944c52acd41c74 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:40.461668 31886 tablet_bootstrap.cc:654] T 445caf2423c1488b8e47474032e68488 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.463864 31888 tablet_bootstrap.cc:492] T 445caf2423c1488b8e47474032e68488 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:40.464021 31888 ts_tablet_manager.cc:1367] T 445caf2423c1488b8e47474032e68488 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.074s user 0.041s sys 0.026s
I20240503 02:12:40.464213 31887 tablet_bootstrap.cc:492] T 445caf2423c1488b8e47474032e68488 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:40.464392 31887 ts_tablet_manager.cc:1367] T 445caf2423c1488b8e47474032e68488 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.069s user 0.028s sys 0.034s
I20240503 02:12:40.464757 31888 raft_consensus.cc:348] T 445caf2423c1488b8e47474032e68488 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:40.464965 31888 raft_consensus.cc:374] T 445caf2423c1488b8e47474032e68488 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.465094 31888 raft_consensus.cc:729] T 445caf2423c1488b8e47474032e68488 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.465302 31889 tablet_bootstrap.cc:654] T a74d38f7d6e5441396944c52acd41c74 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.465271 31887 raft_consensus.cc:348] T 445caf2423c1488b8e47474032e68488 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:40.465342 31888 consensus_queue.cc:260] T 445caf2423c1488b8e47474032e68488 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:40.465531 31887 raft_consensus.cc:374] T 445caf2423c1488b8e47474032e68488 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.466008 31888 ts_tablet_manager.cc:1398] T 445caf2423c1488b8e47474032e68488 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:40.466374 31888 tablet_bootstrap.cc:492] T a74d38f7d6e5441396944c52acd41c74 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:40.465611 31887 raft_consensus.cc:729] T 445caf2423c1488b8e47474032e68488 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.466818 31887 consensus_queue.cc:260] T 445caf2423c1488b8e47474032e68488 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:40.467888 31888 tablet_bootstrap.cc:654] T a74d38f7d6e5441396944c52acd41c74 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.468497 31887 ts_tablet_manager.cc:1398] T 445caf2423c1488b8e47474032e68488 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:12:40.474632 31886 tablet_bootstrap.cc:492] T 445caf2423c1488b8e47474032e68488 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:40.474813 31886 ts_tablet_manager.cc:1367] T 445caf2423c1488b8e47474032e68488 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240503 02:12:40.475641 31888 tablet_bootstrap.cc:492] T a74d38f7d6e5441396944c52acd41c74 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:40.475797 31888 ts_tablet_manager.cc:1367] T a74d38f7d6e5441396944c52acd41c74 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20240503 02:12:40.475930 31886 raft_consensus.cc:348] T 445caf2423c1488b8e47474032e68488 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:40.476193 31886 raft_consensus.cc:374] T 445caf2423c1488b8e47474032e68488 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.476298 31886 raft_consensus.cc:729] T 445caf2423c1488b8e47474032e68488 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.476464 31888 raft_consensus.cc:348] T a74d38f7d6e5441396944c52acd41c74 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:40.476680 31888 raft_consensus.cc:374] T a74d38f7d6e5441396944c52acd41c74 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.476766 31888 raft_consensus.cc:729] T a74d38f7d6e5441396944c52acd41c74 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.476635 31886 consensus_queue.cc:260] T 445caf2423c1488b8e47474032e68488 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:40.477046 31888 consensus_queue.cc:260] T a74d38f7d6e5441396944c52acd41c74 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:40.477718 31888 ts_tablet_manager.cc:1398] T a74d38f7d6e5441396944c52acd41c74 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:12:40.478426 31886 ts_tablet_manager.cc:1398] T 445caf2423c1488b8e47474032e68488 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:40.481328 31889 tablet_bootstrap.cc:492] T a74d38f7d6e5441396944c52acd41c74 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:40.481761 31889 ts_tablet_manager.cc:1367] T a74d38f7d6e5441396944c52acd41c74 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.005s
I20240503 02:12:40.482604 31889 raft_consensus.cc:348] T a74d38f7d6e5441396944c52acd41c74 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:40.482909 31889 raft_consensus.cc:374] T a74d38f7d6e5441396944c52acd41c74 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.483170 31889 raft_consensus.cc:729] T a74d38f7d6e5441396944c52acd41c74 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.483469 31889 consensus_queue.cc:260] T a74d38f7d6e5441396944c52acd41c74 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:40.483922 31889 ts_tablet_manager.cc:1398] T a74d38f7d6e5441396944c52acd41c74 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:12:40.496057 31819 raft_consensus.cc:482] T a592cdf6d1d747c68b5334b90006fa8b P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:40.496199 31819 raft_consensus.cc:504] T a592cdf6d1d747c68b5334b90006fa8b P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:40.497056 31819 leader_election.cc:288] T a592cdf6d1d747c68b5334b90006fa8b P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:12:40.497541 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a592cdf6d1d747c68b5334b90006fa8b" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:12:40.497555 29874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a592cdf6d1d747c68b5334b90006fa8b" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:40.497794 29874 raft_consensus.cc:2379] T a592cdf6d1d747c68b5334b90006fa8b P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a9094240860e43bfb129193f3bfd5be6 in current term 1: Already voted for candidate 155735c7f8b14997ac4c770aee2a9401 in this term.
I20240503 02:12:40.498201 30072 leader_election.cc:302] T a592cdf6d1d747c68b5334b90006fa8b P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6; no voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429
I20240503 02:12:40.498466 31819 raft_consensus.cc:3037] T a592cdf6d1d747c68b5334b90006fa8b P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:40.500355 31819 raft_consensus.cc:2726] T a592cdf6d1d747c68b5334b90006fa8b P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:40.509595 31823 raft_consensus.cc:482] T 445caf2423c1488b8e47474032e68488 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:40.509763 31823 raft_consensus.cc:504] T 445caf2423c1488b8e47474032e68488 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:40.510565 31823 leader_election.cc:288] T 445caf2423c1488b8e47474032e68488 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:40.510991 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445caf2423c1488b8e47474032e68488" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:12:40.511101 29874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445caf2423c1488b8e47474032e68488" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:40.511241 30004 raft_consensus.cc:2454] T 445caf2423c1488b8e47474032e68488 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:40.511292 29874 raft_consensus.cc:2454] T 445caf2423c1488b8e47474032e68488 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:40.511658 30200 leader_election.cc:302] T 445caf2423c1488b8e47474032e68488 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:40.511952 31823 raft_consensus.cc:2781] T 445caf2423c1488b8e47474032e68488 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:40.512073 31823 raft_consensus.cc:482] T 445caf2423c1488b8e47474032e68488 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:40.512168 31823 raft_consensus.cc:3037] T 445caf2423c1488b8e47474032e68488 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:40.514153 31823 raft_consensus.cc:504] T 445caf2423c1488b8e47474032e68488 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:40.514683 31823 leader_election.cc:288] T 445caf2423c1488b8e47474032e68488 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Requested vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:40.515151 29874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445caf2423c1488b8e47474032e68488" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:12:40.515172 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445caf2423c1488b8e47474032e68488" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:12:40.515354 29874 raft_consensus.cc:3037] T 445caf2423c1488b8e47474032e68488 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:40.515354 30004 raft_consensus.cc:3037] T 445caf2423c1488b8e47474032e68488 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:40.518167 29874 raft_consensus.cc:2454] T 445caf2423c1488b8e47474032e68488 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:40.518167 30004 raft_consensus.cc:2454] T 445caf2423c1488b8e47474032e68488 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:40.518616 30202 leader_election.cc:302] T 445caf2423c1488b8e47474032e68488 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:40.518903 31823 raft_consensus.cc:2781] T 445caf2423c1488b8e47474032e68488 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:40.519042 31823 raft_consensus.cc:686] T 445caf2423c1488b8e47474032e68488 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Becoming Leader. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Running, Role: LEADER
I20240503 02:12:40.519300 31823 consensus_queue.cc:237] T 445caf2423c1488b8e47474032e68488 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:40.522370 29729 catalog_manager.cc:5484] T 445caf2423c1488b8e47474032e68488 P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4). New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:12:40.548629 31733 raft_consensus.cc:482] T a74d38f7d6e5441396944c52acd41c74 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:40.548806 31733 raft_consensus.cc:504] T a74d38f7d6e5441396944c52acd41c74 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:40.549644 31733 leader_election.cc:288] T a74d38f7d6e5441396944c52acd41c74 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:12:40.550045 30136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a74d38f7d6e5441396944c52acd41c74" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:12:40.550154 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a74d38f7d6e5441396944c52acd41c74" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:12:40.550285 30136 raft_consensus.cc:2454] T a74d38f7d6e5441396944c52acd41c74 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:12:40.550384 30266 raft_consensus.cc:2454] T a74d38f7d6e5441396944c52acd41c74 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:12:40.550767 29808 leader_election.cc:302] T a74d38f7d6e5441396944c52acd41c74 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:40.551029 31733 raft_consensus.cc:2781] T a74d38f7d6e5441396944c52acd41c74 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:40.551155 31733 raft_consensus.cc:482] T a74d38f7d6e5441396944c52acd41c74 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:40.551249 31733 raft_consensus.cc:3037] T a74d38f7d6e5441396944c52acd41c74 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:40.553195 31733 raft_consensus.cc:504] T a74d38f7d6e5441396944c52acd41c74 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:40.553740 31733 leader_election.cc:288] T a74d38f7d6e5441396944c52acd41c74 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Requested vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:12:40.554143 30136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a74d38f7d6e5441396944c52acd41c74" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:12:40.554190 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a74d38f7d6e5441396944c52acd41c74" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:12:40.554327 30136 raft_consensus.cc:3037] T a74d38f7d6e5441396944c52acd41c74 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:40.554353 30266 raft_consensus.cc:3037] T a74d38f7d6e5441396944c52acd41c74 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:40.556190 30266 raft_consensus.cc:2454] T a74d38f7d6e5441396944c52acd41c74 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:12:40.556237 30136 raft_consensus.cc:2454] T a74d38f7d6e5441396944c52acd41c74 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:12:40.556540 29809 leader_election.cc:302] T a74d38f7d6e5441396944c52acd41c74 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:40.556813 31733 raft_consensus.cc:2781] T a74d38f7d6e5441396944c52acd41c74 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:40.556955 31733 raft_consensus.cc:686] T a74d38f7d6e5441396944c52acd41c74 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Becoming Leader. State: Replica: 16440e365430405fb10366ab70e2b429, State: Running, Role: LEADER
I20240503 02:12:40.557235 31733 consensus_queue.cc:237] T a74d38f7d6e5441396944c52acd41c74 P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:40.560400 29729 catalog_manager.cc:5484] T a74d38f7d6e5441396944c52acd41c74 P 16440e365430405fb10366ab70e2b429 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16440e365430405fb10366ab70e2b429 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:12:40.927783 31826 consensus_queue.cc:1035] T a592cdf6d1d747c68b5334b90006fa8b P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:40.933903 31328 consensus_queue.cc:1035] T a592cdf6d1d747c68b5334b90006fa8b P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:40.946246 31765 consensus_queue.cc:1035] T 445caf2423c1488b8e47474032e68488 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:40.946771 31902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:40.947165 31902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 02:12:40.950996 31765 consensus_queue.cc:1035] T 445caf2423c1488b8e47474032e68488 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:40.958530 31902 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:12:40.974242 31733 consensus_queue.cc:1035] T a74d38f7d6e5441396944c52acd41c74 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:40.980175 31815 consensus_queue.cc:1035] T a74d38f7d6e5441396944c52acd41c74 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_12'
INSERT report
rows total: 2000
time total: 217.649 ms
time per row: 0.108824 ms
I20240503 02:12:41.335253 29726 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalance_test_table_13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:41.344563 30114 tablet_service.cc:1467] Processing CreateTablet for tablet 6bf84757d1fb44689da830a82e857787 (DEFAULT_TABLE table=rebalance_test_table_13 [id=df07c1fd4e654fea9a59ea9a0ea79595]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.345136 30114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bf84757d1fb44689da830a82e857787. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.346330 29985 tablet_service.cc:1467] Processing CreateTablet for tablet db0756f0f331402aaef0583e1f9e5318 (DEFAULT_TABLE table=rebalance_test_table_13 [id=df07c1fd4e654fea9a59ea9a0ea79595]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.346457 30246 tablet_service.cc:1467] Processing CreateTablet for tablet 6bf84757d1fb44689da830a82e857787 (DEFAULT_TABLE table=rebalance_test_table_13 [id=df07c1fd4e654fea9a59ea9a0ea79595]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.346798 29985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db0756f0f331402aaef0583e1f9e5318. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.346850 30246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bf84757d1fb44689da830a82e857787. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.347353 29853 tablet_service.cc:1467] Processing CreateTablet for tablet e61579e74a1a4a9dbd9a09f9b34d9c8d (DEFAULT_TABLE table=rebalance_test_table_13 [id=df07c1fd4e654fea9a59ea9a0ea79595]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.347752 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e61579e74a1a4a9dbd9a09f9b34d9c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.350136 29986 tablet_service.cc:1467] Processing CreateTablet for tablet e61579e74a1a4a9dbd9a09f9b34d9c8d (DEFAULT_TABLE table=rebalance_test_table_13 [id=df07c1fd4e654fea9a59ea9a0ea79595]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.352553 29986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e61579e74a1a4a9dbd9a09f9b34d9c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.353129 29984 tablet_service.cc:1467] Processing CreateTablet for tablet 6bf84757d1fb44689da830a82e857787 (DEFAULT_TABLE table=rebalance_test_table_13 [id=df07c1fd4e654fea9a59ea9a0ea79595]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.353483 29984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bf84757d1fb44689da830a82e857787. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.354008 30115 tablet_service.cc:1467] Processing CreateTablet for tablet db0756f0f331402aaef0583e1f9e5318 (DEFAULT_TABLE table=rebalance_test_table_13 [id=df07c1fd4e654fea9a59ea9a0ea79595]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.354382 30115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db0756f0f331402aaef0583e1f9e5318. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.354751 30244 tablet_service.cc:1467] Processing CreateTablet for tablet db0756f0f331402aaef0583e1f9e5318 (DEFAULT_TABLE table=rebalance_test_table_13 [id=df07c1fd4e654fea9a59ea9a0ea79595]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.355149 30244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db0756f0f331402aaef0583e1f9e5318. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.354050 30116 tablet_service.cc:1467] Processing CreateTablet for tablet e61579e74a1a4a9dbd9a09f9b34d9c8d (DEFAULT_TABLE table=rebalance_test_table_13 [id=df07c1fd4e654fea9a59ea9a0ea79595]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:41.358808 30116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e61579e74a1a4a9dbd9a09f9b34d9c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:41.366672 31967 tablet_bootstrap.cc:492] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:41.371129 31969 tablet_bootstrap.cc:492] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:41.372556 31967 tablet_bootstrap.cc:654] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:41.373010 31969 tablet_bootstrap.cc:654] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:41.374616 31968 tablet_bootstrap.cc:492] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:41.374742 31966 tablet_bootstrap.cc:492] T 6bf84757d1fb44689da830a82e857787 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:41.376641 31968 tablet_bootstrap.cc:654] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:41.381769 31966 tablet_bootstrap.cc:654] T 6bf84757d1fb44689da830a82e857787 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:41.386060 31967 tablet_bootstrap.cc:492] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:41.386229 31969 tablet_bootstrap.cc:492] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:41.386281 31967 ts_tablet_manager.cc:1367] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.004s
I20240503 02:12:41.386482 31969 ts_tablet_manager.cc:1367] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.004s
I20240503 02:12:41.387235 31966 tablet_bootstrap.cc:492] T 6bf84757d1fb44689da830a82e857787 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:41.387074 31967 raft_consensus.cc:348] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:41.387450 31966 ts_tablet_manager.cc:1367] T 6bf84757d1fb44689da830a82e857787 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240503 02:12:41.387351 31969 raft_consensus.cc:348] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:41.387432 31967 raft_consensus.cc:374] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:41.387579 31967 raft_consensus.cc:729] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:41.387588 31969 raft_consensus.cc:374] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:41.387676 31969 raft_consensus.cc:729] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:41.387892 31967 consensus_queue.cc:260] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:41.387992 31969 consensus_queue.cc:260] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:41.388393 31966 raft_consensus.cc:348] T 6bf84757d1fb44689da830a82e857787 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:41.388651 31969 ts_tablet_manager.cc:1398] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:41.388645 31966 raft_consensus.cc:374] T 6bf84757d1fb44689da830a82e857787 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:41.388752 31966 raft_consensus.cc:729] T 6bf84757d1fb44689da830a82e857787 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:41.389871 31969 tablet_bootstrap.cc:492] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:41.389067 31966 consensus_queue.cc:260] T 6bf84757d1fb44689da830a82e857787 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:41.391525 31969 tablet_bootstrap.cc:654] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:41.391980 31966 ts_tablet_manager.cc:1398] T 6bf84757d1fb44689da830a82e857787 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:12:41.394805 31969 tablet_bootstrap.cc:492] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:41.394976 31969 ts_tablet_manager.cc:1367] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240503 02:12:41.395689 31967 ts_tablet_manager.cc:1398] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240503 02:12:41.397121 31968 tablet_bootstrap.cc:492] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:41.397310 31968 ts_tablet_manager.cc:1367] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.003s
I20240503 02:12:41.397507 31966 tablet_bootstrap.cc:492] T db0756f0f331402aaef0583e1f9e5318 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:41.398124 31968 raft_consensus.cc:348] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:41.398363 31968 raft_consensus.cc:374] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:41.398466 31968 raft_consensus.cc:729] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:41.398694 31968 consensus_queue.cc:260] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:41.400545 31968 ts_tablet_manager.cc:1398] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:41.401291 31967 tablet_bootstrap.cc:492] T db0756f0f331402aaef0583e1f9e5318 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:41.401155 31969 raft_consensus.cc:348] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:41.401499 31969 raft_consensus.cc:374] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:41.401633 31969 raft_consensus.cc:729] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:41.402776 31966 tablet_bootstrap.cc:654] T db0756f0f331402aaef0583e1f9e5318 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:41.403009 31969 consensus_queue.cc:260] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:41.403154 31967 tablet_bootstrap.cc:654] T db0756f0f331402aaef0583e1f9e5318 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:41.405580 31969 ts_tablet_manager.cc:1398] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 02:12:41.405972 31969 tablet_bootstrap.cc:492] T 6bf84757d1fb44689da830a82e857787 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:41.408157 31969 tablet_bootstrap.cc:654] T 6bf84757d1fb44689da830a82e857787 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:41.410097 31966 tablet_bootstrap.cc:492] T db0756f0f331402aaef0583e1f9e5318 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:41.410259 31966 ts_tablet_manager.cc:1367] T db0756f0f331402aaef0583e1f9e5318 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.004s
I20240503 02:12:41.410413 31967 tablet_bootstrap.cc:492] T db0756f0f331402aaef0583e1f9e5318 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:41.410555 31967 ts_tablet_manager.cc:1367] T db0756f0f331402aaef0583e1f9e5318 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.006s
I20240503 02:12:41.410926 31966 raft_consensus.cc:348] T db0756f0f331402aaef0583e1f9e5318 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:41.411114 31966 raft_consensus.cc:374] T db0756f0f331402aaef0583e1f9e5318 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:41.411091 31967 raft_consensus.cc:348] T db0756f0f331402aaef0583e1f9e5318 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:41.411195 31966 raft_consensus.cc:729] T db0756f0f331402aaef0583e1f9e5318 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:41.411262 31967 raft_consensus.cc:374] T db0756f0f331402aaef0583e1f9e5318 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:41.411329 31967 raft_consensus.cc:729] T db0756f0f331402aaef0583e1f9e5318 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:41.411443 31966 consensus_queue.cc:260] T db0756f0f331402aaef0583e1f9e5318 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:41.411983 31966 ts_tablet_manager.cc:1398] T db0756f0f331402aaef0583e1f9e5318 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:12:41.411618 31967 consensus_queue.cc:260] T db0756f0f331402aaef0583e1f9e5318 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:41.412176 31969 tablet_bootstrap.cc:492] T 6bf84757d1fb44689da830a82e857787 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:41.412302 31969 ts_tablet_manager.cc:1367] T 6bf84757d1fb44689da830a82e857787 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 02:12:41.413213 31969 raft_consensus.cc:348] T 6bf84757d1fb44689da830a82e857787 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:41.413409 31969 raft_consensus.cc:374] T 6bf84757d1fb44689da830a82e857787 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:41.413499 31969 raft_consensus.cc:729] T 6bf84757d1fb44689da830a82e857787 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:41.413661 31967 ts_tablet_manager.cc:1398] T db0756f0f331402aaef0583e1f9e5318 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:12:41.413712 31969 consensus_queue.cc:260] T 6bf84757d1fb44689da830a82e857787 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:41.414206 31969 ts_tablet_manager.cc:1398] T 6bf84757d1fb44689da830a82e857787 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:41.414745 31966 tablet_bootstrap.cc:492] T e61579e74a1a4a9dbd9a09f9b34d9c8d P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:41.416338 31966 tablet_bootstrap.cc:654] T e61579e74a1a4a9dbd9a09f9b34d9c8d P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:41.418059 31966 tablet_bootstrap.cc:492] T e61579e74a1a4a9dbd9a09f9b34d9c8d P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:41.418188 31966 ts_tablet_manager.cc:1367] T e61579e74a1a4a9dbd9a09f9b34d9c8d P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 02:12:41.418820 31966 raft_consensus.cc:348] T e61579e74a1a4a9dbd9a09f9b34d9c8d P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:41.418968 31966 raft_consensus.cc:374] T e61579e74a1a4a9dbd9a09f9b34d9c8d P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:41.419064 31966 raft_consensus.cc:729] T e61579e74a1a4a9dbd9a09f9b34d9c8d P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:41.419262 31966 consensus_queue.cc:260] T e61579e74a1a4a9dbd9a09f9b34d9c8d P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:41.419719 31966 ts_tablet_manager.cc:1398] T e61579e74a1a4a9dbd9a09f9b34d9c8d P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:12:41.468878 31328 raft_consensus.cc:482] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:41.469061 31328 raft_consensus.cc:504] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:41.469683 31328 leader_election.cc:288] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:41.470211 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61579e74a1a4a9dbd9a09f9b34d9c8d" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:41.470232 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61579e74a1a4a9dbd9a09f9b34d9c8d" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:12:41.470463 29875 raft_consensus.cc:2454] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:12:41.470444 30134 raft_consensus.cc:2454] T e61579e74a1a4a9dbd9a09f9b34d9c8d P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:12:41.470911 29942 leader_election.cc:302] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429; no voters:
I20240503 02:12:41.471168 31328 raft_consensus.cc:2781] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:41.471297 31328 raft_consensus.cc:482] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:41.471395 31328 raft_consensus.cc:3037] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:41.473419 31328 raft_consensus.cc:504] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:41.473953 31328 leader_election.cc:288] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Requested vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:41.474313 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61579e74a1a4a9dbd9a09f9b34d9c8d" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:12:41.474435 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61579e74a1a4a9dbd9a09f9b34d9c8d" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:12:41.474558 29875 raft_consensus.cc:3037] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:41.474639 30134 raft_consensus.cc:3037] T e61579e74a1a4a9dbd9a09f9b34d9c8d P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:41.476586 29875 raft_consensus.cc:2454] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:12:41.476611 30134 raft_consensus.cc:2454] T e61579e74a1a4a9dbd9a09f9b34d9c8d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:12:41.476990 29942 leader_election.cc:302] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429; no voters:
I20240503 02:12:41.477319 31328 raft_consensus.cc:2781] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:41.477705 31328 raft_consensus.cc:686] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Becoming Leader. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Running, Role: LEADER
I20240503 02:12:41.477958 31328 consensus_queue.cc:237] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:41.480885 29729 catalog_manager.cc:5484] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: term changed from 0 to 1, leader changed from <none> to 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2). New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:12:41.515954 31328 raft_consensus.cc:482] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:41.516149 31328 raft_consensus.cc:504] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:41.516805 31328 leader_election.cc:288] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:41.517218 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db0756f0f331402aaef0583e1f9e5318" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:12:41.517261 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db0756f0f331402aaef0583e1f9e5318" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:12:41.517465 30266 raft_consensus.cc:2454] T db0756f0f331402aaef0583e1f9e5318 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:12:41.517489 30134 raft_consensus.cc:2454] T db0756f0f331402aaef0583e1f9e5318 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:12:41.517817 29940 leader_election.cc:302] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:41.518076 31328 raft_consensus.cc:2781] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:41.518170 31328 raft_consensus.cc:482] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:41.518263 31328 raft_consensus.cc:3037] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:41.520325 31328 raft_consensus.cc:504] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:41.520951 31328 leader_election.cc:288] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Requested vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:41.521286 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db0756f0f331402aaef0583e1f9e5318" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:12:41.521363 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db0756f0f331402aaef0583e1f9e5318" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:12:41.521481 30266 raft_consensus.cc:3037] T db0756f0f331402aaef0583e1f9e5318 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:41.521574 30134 raft_consensus.cc:3037] T db0756f0f331402aaef0583e1f9e5318 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:41.523418 30134 raft_consensus.cc:2454] T db0756f0f331402aaef0583e1f9e5318 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:12:41.523422 30266 raft_consensus.cc:2454] T db0756f0f331402aaef0583e1f9e5318 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:12:41.523808 29940 leader_election.cc:302] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:41.524078 31328 raft_consensus.cc:2781] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:41.524227 31328 raft_consensus.cc:686] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Becoming Leader. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Running, Role: LEADER
I20240503 02:12:41.524471 31328 consensus_queue.cc:237] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:41.527643 29726 catalog_manager.cc:5484] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: term changed from 0 to 1, leader changed from <none> to 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2). New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:12:41.556900 31823 raft_consensus.cc:482] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:41.557135 31823 raft_consensus.cc:504] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:41.557871 31823 leader_election.cc:288] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:12:41.558401 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf84757d1fb44689da830a82e857787" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:12:41.558720 30004 raft_consensus.cc:2454] T 6bf84757d1fb44689da830a82e857787 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:41.558970 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf84757d1fb44689da830a82e857787" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:12:41.559217 30200 leader_election.cc:302] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:41.559274 30134 raft_consensus.cc:2454] T 6bf84757d1fb44689da830a82e857787 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:41.559512 31823 raft_consensus.cc:2781] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:41.559609 31823 raft_consensus.cc:482] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:41.559728 31823 raft_consensus.cc:3037] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:41.561714 31823 raft_consensus.cc:504] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:41.562224 31823 leader_election.cc:288] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Requested vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:12:41.562602 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf84757d1fb44689da830a82e857787" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:12:41.562669 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf84757d1fb44689da830a82e857787" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:12:41.562806 30134 raft_consensus.cc:3037] T 6bf84757d1fb44689da830a82e857787 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:41.562860 30004 raft_consensus.cc:3037] T 6bf84757d1fb44689da830a82e857787 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:41.565699 30004 raft_consensus.cc:2454] T 6bf84757d1fb44689da830a82e857787 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:41.566098 30200 leader_election.cc:302] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:41.566123 30134 raft_consensus.cc:2454] T 6bf84757d1fb44689da830a82e857787 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:41.566324 31823 raft_consensus.cc:2781] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:41.566514 31823 raft_consensus.cc:686] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Becoming Leader. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Running, Role: LEADER
I20240503 02:12:41.566789 31823 consensus_queue.cc:237] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:41.569818 29726 catalog_manager.cc:5484] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4). New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:12:41.860975 31328 consensus_queue.cc:1035] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:41.871291 31328 consensus_queue.cc:1035] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:41.974321 31982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:41.974637 31982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:41.982292 31982 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_13'
I20240503 02:12:42.069386 31823 consensus_queue.cc:1035] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:42.074332 31958 consensus_queue.cc:1035] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:42.101142 31328 consensus_queue.cc:1035] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:42.106405 31826 consensus_queue.cc:1035] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:42.312603 29942 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.1:36719, user_credentials={real_user=slave}} blocked reactor thread for 128130us
INSERT report
rows total: 2000
time total: 293.959 ms
time per row: 0.14698 ms
-----------------------------------------------------
Suppressions used:
count bytes template
16 4220 libcrypto.so.1.1
-----------------------------------------------------
I20240503 02:12:43.389227 29734 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalance_test_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:43.398121 30246 tablet_service.cc:1467] Processing CreateTablet for tablet f9dc35fb35834c5d9cb5cea546eccbf8 (DEFAULT_TABLE table=rebalance_test_table_14 [id=079f836c26424ee4adc122852f327cc7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.398645 30246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9dc35fb35834c5d9cb5cea546eccbf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.398844 29853 tablet_service.cc:1467] Processing CreateTablet for tablet f9dc35fb35834c5d9cb5cea546eccbf8 (DEFAULT_TABLE table=rebalance_test_table_14 [id=079f836c26424ee4adc122852f327cc7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.399245 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9dc35fb35834c5d9cb5cea546eccbf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.400110 30244 tablet_service.cc:1467] Processing CreateTablet for tablet db3887ced8964b8196a78a55a1190af4 (DEFAULT_TABLE table=rebalance_test_table_14 [id=079f836c26424ee4adc122852f327cc7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.400269 30245 tablet_service.cc:1467] Processing CreateTablet for tablet ae187b76f8834ba0a7a8b1a3bf33e8e4 (DEFAULT_TABLE table=rebalance_test_table_14 [id=079f836c26424ee4adc122852f327cc7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.400498 30244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db3887ced8964b8196a78a55a1190af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.400754 30245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae187b76f8834ba0a7a8b1a3bf33e8e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.401546 29986 tablet_service.cc:1467] Processing CreateTablet for tablet f9dc35fb35834c5d9cb5cea546eccbf8 (DEFAULT_TABLE table=rebalance_test_table_14 [id=079f836c26424ee4adc122852f327cc7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.401991 29986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9dc35fb35834c5d9cb5cea546eccbf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.405941 29855 tablet_service.cc:1467] Processing CreateTablet for tablet ae187b76f8834ba0a7a8b1a3bf33e8e4 (DEFAULT_TABLE table=rebalance_test_table_14 [id=079f836c26424ee4adc122852f327cc7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.406092 29854 tablet_service.cc:1467] Processing CreateTablet for tablet db3887ced8964b8196a78a55a1190af4 (DEFAULT_TABLE table=rebalance_test_table_14 [id=079f836c26424ee4adc122852f327cc7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.406327 29855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae187b76f8834ba0a7a8b1a3bf33e8e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.407912 30116 tablet_service.cc:1467] Processing CreateTablet for tablet db3887ced8964b8196a78a55a1190af4 (DEFAULT_TABLE table=rebalance_test_table_14 [id=079f836c26424ee4adc122852f327cc7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.408350 30116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db3887ced8964b8196a78a55a1190af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.401546 29984 tablet_service.cc:1467] Processing CreateTablet for tablet ae187b76f8834ba0a7a8b1a3bf33e8e4 (DEFAULT_TABLE table=rebalance_test_table_14 [id=079f836c26424ee4adc122852f327cc7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.413710 29854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db3887ced8964b8196a78a55a1190af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.416242 32053 tablet_bootstrap.cc:492] T f9dc35fb35834c5d9cb5cea546eccbf8 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:43.418380 29984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae187b76f8834ba0a7a8b1a3bf33e8e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.420219 32053 tablet_bootstrap.cc:654] T f9dc35fb35834c5d9cb5cea546eccbf8 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.424496 32053 tablet_bootstrap.cc:492] T f9dc35fb35834c5d9cb5cea546eccbf8 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:43.425817 32053 ts_tablet_manager.cc:1367] T f9dc35fb35834c5d9cb5cea546eccbf8 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20240503 02:12:43.427510 32053 raft_consensus.cc:348] T f9dc35fb35834c5d9cb5cea546eccbf8 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:43.428637 32053 raft_consensus.cc:374] T f9dc35fb35834c5d9cb5cea546eccbf8 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.428928 32053 raft_consensus.cc:729] T f9dc35fb35834c5d9cb5cea546eccbf8 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.429401 32053 consensus_queue.cc:260] T f9dc35fb35834c5d9cb5cea546eccbf8 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:43.429595 32054 tablet_bootstrap.cc:492] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:43.431108 32053 ts_tablet_manager.cc:1398] T f9dc35fb35834c5d9cb5cea546eccbf8 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 02:12:43.431473 32053 tablet_bootstrap.cc:492] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:43.432868 32054 tablet_bootstrap.cc:654] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.433245 32053 tablet_bootstrap.cc:654] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.438234 32054 tablet_bootstrap.cc:492] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:43.438900 32054 ts_tablet_manager.cc:1367] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240503 02:12:43.440018 32054 raft_consensus.cc:348] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:43.441813 32054 raft_consensus.cc:374] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.442134 32054 raft_consensus.cc:729] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.442579 32054 consensus_queue.cc:260] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:43.443917 32054 ts_tablet_manager.cc:1398] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:12:43.445691 32054 tablet_bootstrap.cc:492] T db3887ced8964b8196a78a55a1190af4 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:43.448809 32054 tablet_bootstrap.cc:654] T db3887ced8964b8196a78a55a1190af4 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.456346 32054 tablet_bootstrap.cc:492] T db3887ced8964b8196a78a55a1190af4 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:43.457051 32054 ts_tablet_manager.cc:1367] T db3887ced8964b8196a78a55a1190af4 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20240503 02:12:43.458698 32054 raft_consensus.cc:348] T db3887ced8964b8196a78a55a1190af4 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:43.460397 32054 raft_consensus.cc:374] T db3887ced8964b8196a78a55a1190af4 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.460700 32054 raft_consensus.cc:729] T db3887ced8964b8196a78a55a1190af4 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.461236 32054 consensus_queue.cc:260] T db3887ced8964b8196a78a55a1190af4 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:43.462715 32054 ts_tablet_manager.cc:1398] T db3887ced8964b8196a78a55a1190af4 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 02:12:43.463743 32054 tablet_bootstrap.cc:492] T f9dc35fb35834c5d9cb5cea546eccbf8 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:43.467308 32054 tablet_bootstrap.cc:654] T f9dc35fb35834c5d9cb5cea546eccbf8 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.477082 32054 tablet_bootstrap.cc:492] T f9dc35fb35834c5d9cb5cea546eccbf8 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:43.477946 32054 ts_tablet_manager.cc:1367] T f9dc35fb35834c5d9cb5cea546eccbf8 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.002s
I20240503 02:12:43.479225 32054 raft_consensus.cc:348] T f9dc35fb35834c5d9cb5cea546eccbf8 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:43.480777 32054 raft_consensus.cc:374] T f9dc35fb35834c5d9cb5cea546eccbf8 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.481081 32054 raft_consensus.cc:729] T f9dc35fb35834c5d9cb5cea546eccbf8 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.481693 32054 consensus_queue.cc:260] T f9dc35fb35834c5d9cb5cea546eccbf8 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:43.483088 32054 ts_tablet_manager.cc:1398] T f9dc35fb35834c5d9cb5cea546eccbf8 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 02:12:43.495260 32056 tablet_bootstrap.cc:492] T f9dc35fb35834c5d9cb5cea546eccbf8 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:43.495859 32057 tablet_bootstrap.cc:492] T db3887ced8964b8196a78a55a1190af4 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:43.496109 32053 tablet_bootstrap.cc:492] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:43.496315 32053 ts_tablet_manager.cc:1367] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.065s user 0.029s sys 0.029s
I20240503 02:12:43.497570 32053 raft_consensus.cc:348] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:43.497756 32057 tablet_bootstrap.cc:654] T db3887ced8964b8196a78a55a1190af4 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.497926 32053 raft_consensus.cc:374] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.498076 32053 raft_consensus.cc:729] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.498514 32053 consensus_queue.cc:260] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:43.499467 32053 ts_tablet_manager.cc:1398] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:43.499903 32053 tablet_bootstrap.cc:492] T db3887ced8964b8196a78a55a1190af4 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:43.502120 32053 tablet_bootstrap.cc:654] T db3887ced8964b8196a78a55a1190af4 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.505355 32056 tablet_bootstrap.cc:654] T f9dc35fb35834c5d9cb5cea546eccbf8 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.506115 32057 tablet_bootstrap.cc:492] T db3887ced8964b8196a78a55a1190af4 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:43.506314 32057 ts_tablet_manager.cc:1367] T db3887ced8964b8196a78a55a1190af4 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240503 02:12:43.507319 32057 raft_consensus.cc:348] T db3887ced8964b8196a78a55a1190af4 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:43.507582 32057 raft_consensus.cc:374] T db3887ced8964b8196a78a55a1190af4 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.507696 32057 raft_consensus.cc:729] T db3887ced8964b8196a78a55a1190af4 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.507979 32057 consensus_queue.cc:260] T db3887ced8964b8196a78a55a1190af4 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:43.508668 32057 ts_tablet_manager.cc:1398] T db3887ced8964b8196a78a55a1190af4 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:43.510008 32053 tablet_bootstrap.cc:492] T db3887ced8964b8196a78a55a1190af4 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:43.510449 32053 ts_tablet_manager.cc:1367] T db3887ced8964b8196a78a55a1190af4 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.003s
I20240503 02:12:43.511291 32053 raft_consensus.cc:348] T db3887ced8964b8196a78a55a1190af4 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:43.511519 32053 raft_consensus.cc:374] T db3887ced8964b8196a78a55a1190af4 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.511632 32053 raft_consensus.cc:729] T db3887ced8964b8196a78a55a1190af4 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.511986 32053 consensus_queue.cc:260] T db3887ced8964b8196a78a55a1190af4 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:43.513214 32053 ts_tablet_manager.cc:1398] T db3887ced8964b8196a78a55a1190af4 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:12:43.514645 32056 tablet_bootstrap.cc:492] T f9dc35fb35834c5d9cb5cea546eccbf8 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:43.514825 32056 ts_tablet_manager.cc:1367] T f9dc35fb35834c5d9cb5cea546eccbf8 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.003s
I20240503 02:12:43.515605 32056 raft_consensus.cc:348] T f9dc35fb35834c5d9cb5cea546eccbf8 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:43.515825 32056 raft_consensus.cc:374] T f9dc35fb35834c5d9cb5cea546eccbf8 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.515921 32056 raft_consensus.cc:729] T f9dc35fb35834c5d9cb5cea546eccbf8 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.516129 32056 consensus_queue.cc:260] T f9dc35fb35834c5d9cb5cea546eccbf8 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:43.516618 32056 ts_tablet_manager.cc:1398] T f9dc35fb35834c5d9cb5cea546eccbf8 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:43.516912 32056 tablet_bootstrap.cc:492] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:43.518626 32056 tablet_bootstrap.cc:654] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.520409 32056 tablet_bootstrap.cc:492] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:43.520638 32056 ts_tablet_manager.cc:1367] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:12:43.521446 32056 raft_consensus.cc:348] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:43.521625 32056 raft_consensus.cc:374] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.521703 32056 raft_consensus.cc:729] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.521922 32056 consensus_queue.cc:260] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:43.522383 32056 ts_tablet_manager.cc:1398] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:43.559813 31958 raft_consensus.cc:482] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:43.560024 31958 raft_consensus.cc:504] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:43.560607 31958 leader_election.cc:288] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:12:43.560982 29873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae187b76f8834ba0a7a8b1a3bf33e8e4" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:43.561122 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae187b76f8834ba0a7a8b1a3bf33e8e4" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:12:43.561231 29873 raft_consensus.cc:2454] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:43.561347 30005 raft_consensus.cc:2454] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:43.561601 30202 leader_election.cc:302] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:43.561885 31958 raft_consensus.cc:2781] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:43.561987 31958 raft_consensus.cc:482] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:43.562091 31958 raft_consensus.cc:3037] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.564092 31958 raft_consensus.cc:504] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:43.564610 31958 leader_election.cc:288] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Requested vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:12:43.565073 29873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae187b76f8834ba0a7a8b1a3bf33e8e4" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:12:43.565079 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae187b76f8834ba0a7a8b1a3bf33e8e4" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:12:43.565254 30005 raft_consensus.cc:3037] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.565254 29873 raft_consensus.cc:3037] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.567989 29873 raft_consensus.cc:2454] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:43.567989 30005 raft_consensus.cc:2454] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:43.568375 30202 leader_election.cc:302] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:43.568676 31958 raft_consensus.cc:2781] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:43.568866 31958 raft_consensus.cc:686] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Becoming Leader. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Running, Role: LEADER
I20240503 02:12:43.569145 31958 consensus_queue.cc:237] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:43.572678 29734 catalog_manager.cc:5484] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4). New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:12:43.621757 32034 raft_consensus.cc:482] T db3887ced8964b8196a78a55a1190af4 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:43.621942 32034 raft_consensus.cc:504] T db3887ced8964b8196a78a55a1190af4 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:43.622552 32034 leader_election.cc:288] T db3887ced8964b8196a78a55a1190af4 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:43.622907 29873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db3887ced8964b8196a78a55a1190af4" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:43.623096 30135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db3887ced8964b8196a78a55a1190af4" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:12:43.623200 29873 raft_consensus.cc:2454] T db3887ced8964b8196a78a55a1190af4 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:43.623420 30135 raft_consensus.cc:2454] T db3887ced8964b8196a78a55a1190af4 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:43.623651 30202 leader_election.cc:302] T db3887ced8964b8196a78a55a1190af4 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:43.623932 32034 raft_consensus.cc:2781] T db3887ced8964b8196a78a55a1190af4 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:43.624032 32034 raft_consensus.cc:482] T db3887ced8964b8196a78a55a1190af4 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:43.624121 32034 raft_consensus.cc:3037] T db3887ced8964b8196a78a55a1190af4 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.626145 32034 raft_consensus.cc:504] T db3887ced8964b8196a78a55a1190af4 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:43.626739 32034 leader_election.cc:288] T db3887ced8964b8196a78a55a1190af4 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Requested vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:43.627168 30135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db3887ced8964b8196a78a55a1190af4" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:12:43.627290 29873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db3887ced8964b8196a78a55a1190af4" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:12:43.627382 30135 raft_consensus.cc:3037] T db3887ced8964b8196a78a55a1190af4 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.627539 29873 raft_consensus.cc:3037] T db3887ced8964b8196a78a55a1190af4 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.629362 30135 raft_consensus.cc:2454] T db3887ced8964b8196a78a55a1190af4 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:43.629443 29873 raft_consensus.cc:2454] T db3887ced8964b8196a78a55a1190af4 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:43.629722 30199 leader_election.cc:302] T db3887ced8964b8196a78a55a1190af4 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:43.629976 32034 raft_consensus.cc:2781] T db3887ced8964b8196a78a55a1190af4 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:43.630103 32034 raft_consensus.cc:686] T db3887ced8964b8196a78a55a1190af4 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Becoming Leader. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Running, Role: LEADER
I20240503 02:12:43.630345 32034 consensus_queue.cc:237] T db3887ced8964b8196a78a55a1190af4 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:43.633477 29734 catalog_manager.cc:5484] T db3887ced8964b8196a78a55a1190af4 P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4). New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:12:43.690721 32061 raft_consensus.cc:482] T f9dc35fb35834c5d9cb5cea546eccbf8 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:43.690881 32061 raft_consensus.cc:504] T f9dc35fb35834c5d9cb5cea546eccbf8 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:43.691572 32061 leader_election.cc:288] T f9dc35fb35834c5d9cb5cea546eccbf8 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:43.692018 29873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9dc35fb35834c5d9cb5cea546eccbf8" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:43.692001 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9dc35fb35834c5d9cb5cea546eccbf8" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:12:43.692226 29873 raft_consensus.cc:2454] T f9dc35fb35834c5d9cb5cea546eccbf8 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:12:43.692226 30265 raft_consensus.cc:2454] T f9dc35fb35834c5d9cb5cea546eccbf8 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:12:43.692580 29942 leader_election.cc:302] T f9dc35fb35834c5d9cb5cea546eccbf8 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429; no voters:
I20240503 02:12:43.692811 32061 raft_consensus.cc:2781] T f9dc35fb35834c5d9cb5cea546eccbf8 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:43.692909 32061 raft_consensus.cc:482] T f9dc35fb35834c5d9cb5cea546eccbf8 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:43.692999 32061 raft_consensus.cc:3037] T f9dc35fb35834c5d9cb5cea546eccbf8 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.694936 32061 raft_consensus.cc:504] T f9dc35fb35834c5d9cb5cea546eccbf8 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:43.695472 32061 leader_election.cc:288] T f9dc35fb35834c5d9cb5cea546eccbf8 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Requested vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:43.695963 29873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9dc35fb35834c5d9cb5cea546eccbf8" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:12:43.696031 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9dc35fb35834c5d9cb5cea546eccbf8" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:12:43.696159 29873 raft_consensus.cc:3037] T f9dc35fb35834c5d9cb5cea546eccbf8 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.696218 30265 raft_consensus.cc:3037] T f9dc35fb35834c5d9cb5cea546eccbf8 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.699014 30265 raft_consensus.cc:2454] T f9dc35fb35834c5d9cb5cea546eccbf8 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:12:43.699012 29873 raft_consensus.cc:2454] T f9dc35fb35834c5d9cb5cea546eccbf8 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:12:43.699426 29940 leader_election.cc:302] T f9dc35fb35834c5d9cb5cea546eccbf8 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:43.699746 32061 raft_consensus.cc:2781] T f9dc35fb35834c5d9cb5cea546eccbf8 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:43.699887 32061 raft_consensus.cc:686] T f9dc35fb35834c5d9cb5cea546eccbf8 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Becoming Leader. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Running, Role: LEADER
I20240503 02:12:43.700161 32061 consensus_queue.cc:237] T f9dc35fb35834c5d9cb5cea546eccbf8 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:43.703274 29734 catalog_manager.cc:5484] T f9dc35fb35834c5d9cb5cea546eccbf8 P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: term changed from 0 to 1, leader changed from <none> to 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2). New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:12:44.070524 31958 consensus_queue.cc:1035] T db3887ced8964b8196a78a55a1190af4 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:44.076941 31958 consensus_queue.cc:1035] T db3887ced8964b8196a78a55a1190af4 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:44.083928 31958 consensus_queue.cc:1035] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:44.088827 32034 consensus_queue.cc:1035] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:44.114867 32070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:44.115322 32070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:44.123348 32070 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_14'
I20240503 02:12:44.169349 32061 consensus_queue.cc:1035] T f9dc35fb35834c5d9cb5cea546eccbf8 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:44.174211 32030 consensus_queue.cc:1035] T f9dc35fb35834c5d9cb5cea546eccbf8 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 230.77 ms
time per row: 0.115385 ms
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240503 02:12:45.473793 29734 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalance_test_table_15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:45.482635 29854 tablet_service.cc:1467] Processing CreateTablet for tablet 63b416f4e004490b8e38a96fd065dfea (DEFAULT_TABLE table=rebalance_test_table_15 [id=6bbf46811ff140d289e600493149f118]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.482600 30116 tablet_service.cc:1467] Processing CreateTablet for tablet 63b416f4e004490b8e38a96fd065dfea (DEFAULT_TABLE table=rebalance_test_table_15 [id=6bbf46811ff140d289e600493149f118]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.483070 29854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63b416f4e004490b8e38a96fd065dfea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.483075 30116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63b416f4e004490b8e38a96fd065dfea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.482954 30114 tablet_service.cc:1467] Processing CreateTablet for tablet 6635aacafed54994965ad48b2df4f163 (DEFAULT_TABLE table=rebalance_test_table_15 [id=6bbf46811ff140d289e600493149f118]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.483389 30114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6635aacafed54994965ad48b2df4f163. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.484493 30245 tablet_service.cc:1467] Processing CreateTablet for tablet 6635aacafed54994965ad48b2df4f163 (DEFAULT_TABLE table=rebalance_test_table_15 [id=6bbf46811ff140d289e600493149f118]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.484931 30245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6635aacafed54994965ad48b2df4f163. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.489470 29986 tablet_service.cc:1467] Processing CreateTablet for tablet 63b416f4e004490b8e38a96fd065dfea (DEFAULT_TABLE table=rebalance_test_table_15 [id=6bbf46811ff140d289e600493149f118]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.489650 29984 tablet_service.cc:1467] Processing CreateTablet for tablet 6635aacafed54994965ad48b2df4f163 (DEFAULT_TABLE table=rebalance_test_table_15 [id=6bbf46811ff140d289e600493149f118]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.489938 29986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63b416f4e004490b8e38a96fd065dfea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.490501 29984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6635aacafed54994965ad48b2df4f163. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.493605 30246 tablet_service.cc:1467] Processing CreateTablet for tablet 1c9200fa969a46e6a48c64293801382b (DEFAULT_TABLE table=rebalance_test_table_15 [id=6bbf46811ff140d289e600493149f118]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.494133 30246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c9200fa969a46e6a48c64293801382b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.496948 30115 tablet_service.cc:1467] Processing CreateTablet for tablet 1c9200fa969a46e6a48c64293801382b (DEFAULT_TABLE table=rebalance_test_table_15 [id=6bbf46811ff140d289e600493149f118]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.497444 30115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c9200fa969a46e6a48c64293801382b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.498008 32142 tablet_bootstrap.cc:492] T 63b416f4e004490b8e38a96fd065dfea P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:45.498862 32143 tablet_bootstrap.cc:492] T 6635aacafed54994965ad48b2df4f163 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:45.499384 29853 tablet_service.cc:1467] Processing CreateTablet for tablet 1c9200fa969a46e6a48c64293801382b (DEFAULT_TABLE table=rebalance_test_table_15 [id=6bbf46811ff140d289e600493149f118]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.499684 32142 tablet_bootstrap.cc:654] T 63b416f4e004490b8e38a96fd065dfea P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:45.499779 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c9200fa969a46e6a48c64293801382b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.504506 32143 tablet_bootstrap.cc:654] T 6635aacafed54994965ad48b2df4f163 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:45.507280 32145 tablet_bootstrap.cc:492] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:45.509203 32144 tablet_bootstrap.cc:492] T 63b416f4e004490b8e38a96fd065dfea P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:45.512989 32144 tablet_bootstrap.cc:654] T 63b416f4e004490b8e38a96fd065dfea P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:45.512226 32145 tablet_bootstrap.cc:654] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:45.515223 32143 tablet_bootstrap.cc:492] T 6635aacafed54994965ad48b2df4f163 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:45.515451 32143 ts_tablet_manager.cc:1367] T 6635aacafed54994965ad48b2df4f163 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.002s
I20240503 02:12:45.516386 32143 raft_consensus.cc:348] T 6635aacafed54994965ad48b2df4f163 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:45.516780 32143 raft_consensus.cc:374] T 6635aacafed54994965ad48b2df4f163 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:45.517004 32143 raft_consensus.cc:729] T 6635aacafed54994965ad48b2df4f163 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:45.517413 32143 consensus_queue.cc:260] T 6635aacafed54994965ad48b2df4f163 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:45.519127 32143 ts_tablet_manager.cc:1398] T 6635aacafed54994965ad48b2df4f163 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:12:45.524765 32144 tablet_bootstrap.cc:492] T 63b416f4e004490b8e38a96fd065dfea P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:45.525002 32144 ts_tablet_manager.cc:1367] T 63b416f4e004490b8e38a96fd065dfea P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240503 02:12:45.525132 32145 tablet_bootstrap.cc:492] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:45.525615 32145 ts_tablet_manager.cc:1367] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.003s
I20240503 02:12:45.526587 32145 raft_consensus.cc:348] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:45.526898 32145 raft_consensus.cc:374] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:45.527050 32145 raft_consensus.cc:729] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:45.527006 32144 raft_consensus.cc:348] T 63b416f4e004490b8e38a96fd065dfea P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:45.527423 32145 consensus_queue.cc:260] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:45.527626 32144 raft_consensus.cc:374] T 63b416f4e004490b8e38a96fd065dfea P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:45.527736 32144 raft_consensus.cc:729] T 63b416f4e004490b8e38a96fd065dfea P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:45.527954 32144 consensus_queue.cc:260] T 63b416f4e004490b8e38a96fd065dfea P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:45.528525 32144 ts_tablet_manager.cc:1398] T 63b416f4e004490b8e38a96fd065dfea P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 02:12:45.528777 32144 tablet_bootstrap.cc:492] T 1c9200fa969a46e6a48c64293801382b P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:45.532160 32144 tablet_bootstrap.cc:654] T 1c9200fa969a46e6a48c64293801382b P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:45.535682 32143 tablet_bootstrap.cc:492] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:45.538153 32143 tablet_bootstrap.cc:654] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:45.538446 32145 ts_tablet_manager.cc:1398] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240503 02:12:45.540805 32143 tablet_bootstrap.cc:492] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:45.540983 32143 ts_tablet_manager.cc:1367] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240503 02:12:45.541805 32143 raft_consensus.cc:348] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:45.542042 32143 raft_consensus.cc:374] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:45.542131 32143 raft_consensus.cc:729] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:45.542369 32143 consensus_queue.cc:260] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:45.542941 32143 ts_tablet_manager.cc:1398] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:45.543969 32144 tablet_bootstrap.cc:492] T 1c9200fa969a46e6a48c64293801382b P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:45.544137 32144 ts_tablet_manager.cc:1367] T 1c9200fa969a46e6a48c64293801382b P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.003s
I20240503 02:12:45.544304 32142 tablet_bootstrap.cc:492] T 63b416f4e004490b8e38a96fd065dfea P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:45.544456 32142 ts_tablet_manager.cc:1367] T 63b416f4e004490b8e38a96fd065dfea P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.047s user 0.008s sys 0.033s
I20240503 02:12:45.544864 32144 raft_consensus.cc:348] T 1c9200fa969a46e6a48c64293801382b P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:45.545107 32144 raft_consensus.cc:374] T 1c9200fa969a46e6a48c64293801382b P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:45.545192 32144 raft_consensus.cc:729] T 1c9200fa969a46e6a48c64293801382b P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:45.545440 32144 consensus_queue.cc:260] T 1c9200fa969a46e6a48c64293801382b P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:45.545486 32142 raft_consensus.cc:348] T 63b416f4e004490b8e38a96fd065dfea P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:45.545799 32142 raft_consensus.cc:374] T 63b416f4e004490b8e38a96fd065dfea P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:45.545923 32142 raft_consensus.cc:729] T 63b416f4e004490b8e38a96fd065dfea P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:45.546011 32144 ts_tablet_manager.cc:1398] T 1c9200fa969a46e6a48c64293801382b P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:12:45.546339 32144 tablet_bootstrap.cc:492] T 6635aacafed54994965ad48b2df4f163 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:45.546228 32142 consensus_queue.cc:260] T 63b416f4e004490b8e38a96fd065dfea P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:45.547101 32142 ts_tablet_manager.cc:1398] T 63b416f4e004490b8e38a96fd065dfea P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:45.547551 32142 tablet_bootstrap.cc:492] T 1c9200fa969a46e6a48c64293801382b P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:45.548003 32144 tablet_bootstrap.cc:654] T 6635aacafed54994965ad48b2df4f163 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:45.549338 32142 tablet_bootstrap.cc:654] T 1c9200fa969a46e6a48c64293801382b P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:45.549846 32145 tablet_bootstrap.cc:492] T 63b416f4e004490b8e38a96fd065dfea P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:45.551275 32145 tablet_bootstrap.cc:654] T 63b416f4e004490b8e38a96fd065dfea P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:45.552661 32144 tablet_bootstrap.cc:492] T 6635aacafed54994965ad48b2df4f163 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:45.552807 32144 ts_tablet_manager.cc:1367] T 6635aacafed54994965ad48b2df4f163 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20240503 02:12:45.553586 32144 raft_consensus.cc:348] T 6635aacafed54994965ad48b2df4f163 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:45.553831 32144 raft_consensus.cc:374] T 6635aacafed54994965ad48b2df4f163 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:45.553932 32144 raft_consensus.cc:729] T 6635aacafed54994965ad48b2df4f163 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:45.554152 32144 consensus_queue.cc:260] T 6635aacafed54994965ad48b2df4f163 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:45.554674 32144 ts_tablet_manager.cc:1398] T 6635aacafed54994965ad48b2df4f163 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:45.556551 32145 tablet_bootstrap.cc:492] T 63b416f4e004490b8e38a96fd065dfea P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:45.556720 32145 ts_tablet_manager.cc:1367] T 63b416f4e004490b8e38a96fd065dfea P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 02:12:45.557569 32145 raft_consensus.cc:348] T 63b416f4e004490b8e38a96fd065dfea P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:45.557829 32145 raft_consensus.cc:374] T 63b416f4e004490b8e38a96fd065dfea P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:45.557909 32145 raft_consensus.cc:729] T 63b416f4e004490b8e38a96fd065dfea P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:45.558284 32145 consensus_queue.cc:260] T 63b416f4e004490b8e38a96fd065dfea P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:45.558538 32137 raft_consensus.cc:482] T 63b416f4e004490b8e38a96fd065dfea P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:45.558653 32137 raft_consensus.cc:504] T 63b416f4e004490b8e38a96fd065dfea P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:45.559268 32137 leader_election.cc:288] T 63b416f4e004490b8e38a96fd065dfea P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:45.559573 30135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b416f4e004490b8e38a96fd065dfea" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:12:45.559806 30135 raft_consensus.cc:2454] T 63b416f4e004490b8e38a96fd065dfea P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:12:45.559818 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b416f4e004490b8e38a96fd065dfea" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:45.560019 29875 raft_consensus.cc:2454] T 63b416f4e004490b8e38a96fd065dfea P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:12:45.560256 29939 leader_election.cc:302] T 63b416f4e004490b8e38a96fd065dfea P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:45.560704 32145 ts_tablet_manager.cc:1398] T 63b416f4e004490b8e38a96fd065dfea P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:12:45.560770 32142 tablet_bootstrap.cc:492] T 1c9200fa969a46e6a48c64293801382b P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:45.560930 32142 ts_tablet_manager.cc:1367] T 1c9200fa969a46e6a48c64293801382b P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.007s
I20240503 02:12:45.560950 32061 raft_consensus.cc:2781] T 63b416f4e004490b8e38a96fd065dfea P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:45.561054 32061 raft_consensus.cc:482] T 63b416f4e004490b8e38a96fd065dfea P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:45.561173 32061 raft_consensus.cc:3037] T 63b416f4e004490b8e38a96fd065dfea P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:45.561754 32142 raft_consensus.cc:348] T 1c9200fa969a46e6a48c64293801382b P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:45.561962 32142 raft_consensus.cc:374] T 1c9200fa969a46e6a48c64293801382b P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:45.562024 32142 raft_consensus.cc:729] T 1c9200fa969a46e6a48c64293801382b P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:45.562266 32142 consensus_queue.cc:260] T 1c9200fa969a46e6a48c64293801382b P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:45.562938 32142 ts_tablet_manager.cc:1398] T 1c9200fa969a46e6a48c64293801382b P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:45.563722 32061 raft_consensus.cc:504] T 63b416f4e004490b8e38a96fd065dfea P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:45.564335 32061 leader_election.cc:288] T 63b416f4e004490b8e38a96fd065dfea P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Requested vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:45.564699 30135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b416f4e004490b8e38a96fd065dfea" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:12:45.564863 30135 raft_consensus.cc:3037] T 63b416f4e004490b8e38a96fd065dfea P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:45.564849 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b416f4e004490b8e38a96fd065dfea" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:12:45.565060 29875 raft_consensus.cc:3037] T 63b416f4e004490b8e38a96fd065dfea P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:45.566913 30135 raft_consensus.cc:2454] T 63b416f4e004490b8e38a96fd065dfea P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:12:45.567253 29939 leader_election.cc:302] T 63b416f4e004490b8e38a96fd065dfea P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:45.567517 32061 raft_consensus.cc:2781] T 63b416f4e004490b8e38a96fd065dfea P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:45.567646 32061 raft_consensus.cc:686] T 63b416f4e004490b8e38a96fd065dfea P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Becoming Leader. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Running, Role: LEADER
I20240503 02:12:45.567873 29875 raft_consensus.cc:2454] T 63b416f4e004490b8e38a96fd065dfea P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:12:45.567924 32061 consensus_queue.cc:237] T 63b416f4e004490b8e38a96fd065dfea P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:45.571125 29734 catalog_manager.cc:5484] T 63b416f4e004490b8e38a96fd065dfea P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: term changed from 0 to 1, leader changed from <none> to 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2). New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:12:45.571381 32061 raft_consensus.cc:482] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:45.571539 32061 raft_consensus.cc:504] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:45.572114 32061 leader_election.cc:288] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:12:45.572407 30135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6635aacafed54994965ad48b2df4f163" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:12:45.572644 30135 raft_consensus.cc:2454] T 6635aacafed54994965ad48b2df4f163 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:12:45.572649 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6635aacafed54994965ad48b2df4f163" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:12:45.572877 30266 raft_consensus.cc:2454] T 6635aacafed54994965ad48b2df4f163 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:12:45.573002 29939 leader_election.cc:302] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:45.573278 32061 raft_consensus.cc:2781] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:45.573400 32061 raft_consensus.cc:482] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:45.573522 32061 raft_consensus.cc:3037] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:45.575873 32061 raft_consensus.cc:504] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:45.576345 32061 leader_election.cc:288] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Requested vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:12:45.576946 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6635aacafed54994965ad48b2df4f163" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:12:45.577152 30266 raft_consensus.cc:3037] T 6635aacafed54994965ad48b2df4f163 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:45.577333 30135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6635aacafed54994965ad48b2df4f163" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:12:45.577564 30135 raft_consensus.cc:3037] T 6635aacafed54994965ad48b2df4f163 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:45.580194 30266 raft_consensus.cc:2454] T 6635aacafed54994965ad48b2df4f163 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:12:45.580304 30135 raft_consensus.cc:2454] T 6635aacafed54994965ad48b2df4f163 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:12:45.580585 29940 leader_election.cc:302] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:45.580927 32061 raft_consensus.cc:2781] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:45.581112 32061 raft_consensus.cc:686] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Becoming Leader. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Running, Role: LEADER
I20240503 02:12:45.581389 32061 consensus_queue.cc:237] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:45.584525 29734 catalog_manager.cc:5484] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: term changed from 0 to 1, leader changed from <none> to 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2). New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:12:45.727432 32123 raft_consensus.cc:482] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:45.727655 32123 raft_consensus.cc:504] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:45.729053 32123 leader_election.cc:288] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:45.729346 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c9200fa969a46e6a48c64293801382b" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:45.729405 30135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c9200fa969a46e6a48c64293801382b" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:12:45.729605 29875 raft_consensus.cc:2454] T 1c9200fa969a46e6a48c64293801382b P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:45.729667 30135 raft_consensus.cc:2454] T 1c9200fa969a46e6a48c64293801382b P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:45.730031 30199 leader_election.cc:302] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:45.730295 32123 raft_consensus.cc:2781] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:45.730409 32123 raft_consensus.cc:482] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:45.730496 32123 raft_consensus.cc:3037] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:45.732482 32123 raft_consensus.cc:504] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:45.733081 32123 leader_election.cc:288] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Requested vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:45.733457 30135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c9200fa969a46e6a48c64293801382b" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:12:45.733636 30135 raft_consensus.cc:3037] T 1c9200fa969a46e6a48c64293801382b P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:45.733580 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c9200fa969a46e6a48c64293801382b" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:12:45.733791 29875 raft_consensus.cc:3037] T 1c9200fa969a46e6a48c64293801382b P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:45.735603 30135 raft_consensus.cc:2454] T 1c9200fa969a46e6a48c64293801382b P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:45.735771 29875 raft_consensus.cc:2454] T 1c9200fa969a46e6a48c64293801382b P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:45.736044 30199 leader_election.cc:302] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:45.736325 32123 raft_consensus.cc:2781] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:45.736452 32123 raft_consensus.cc:686] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Becoming Leader. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Running, Role: LEADER
I20240503 02:12:45.736725 32123 consensus_queue.cc:237] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:45.740043 29734 catalog_manager.cc:5484] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4). New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:12:45.989197 32061 consensus_queue.cc:1035] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:45.994078 32061 consensus_queue.cc:1035] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:46.049149 32061 consensus_queue.cc:1035] T 63b416f4e004490b8e38a96fd065dfea P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:46.054930 32061 consensus_queue.cc:1035] T 63b416f4e004490b8e38a96fd065dfea P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:46.189625 32160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:46.189996 32160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:46.197556 32160 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_15'
I20240503 02:12:46.280316 32158 consensus_queue.cc:1035] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:46.285419 32123 consensus_queue.cc:1035] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 253.677 ms
time per row: 0.126838 ms
I20240503 02:12:46.625285 29734 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalance_test_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:46.634577 29853 tablet_service.cc:1467] Processing CreateTablet for tablet ed32c167c0d74ee588b59f240d2e5b24 (DEFAULT_TABLE table=rebalance_test_table_16 [id=0cc2eea0038e4971a3d7119cbaabc735]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.635135 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed32c167c0d74ee588b59f240d2e5b24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.635246 30115 tablet_service.cc:1467] Processing CreateTablet for tablet ed32c167c0d74ee588b59f240d2e5b24 (DEFAULT_TABLE table=rebalance_test_table_16 [id=0cc2eea0038e4971a3d7119cbaabc735]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.635313 29854 tablet_service.cc:1467] Processing CreateTablet for tablet 4de7cb0be278476481d01083add5c978 (DEFAULT_TABLE table=rebalance_test_table_16 [id=0cc2eea0038e4971a3d7119cbaabc735]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.635639 30115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed32c167c0d74ee588b59f240d2e5b24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.635744 29854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4de7cb0be278476481d01083add5c978. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.638581 29984 tablet_service.cc:1467] Processing CreateTablet for tablet 4de7cb0be278476481d01083add5c978 (DEFAULT_TABLE table=rebalance_test_table_16 [id=0cc2eea0038e4971a3d7119cbaabc735]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.638976 29984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4de7cb0be278476481d01083add5c978. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.639966 30246 tablet_service.cc:1467] Processing CreateTablet for tablet 4de7cb0be278476481d01083add5c978 (DEFAULT_TABLE table=rebalance_test_table_16 [id=0cc2eea0038e4971a3d7119cbaabc735]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.640033 30245 tablet_service.cc:1467] Processing CreateTablet for tablet c3f04e7716c54d6090a00d7a6b771b64 (DEFAULT_TABLE table=rebalance_test_table_16 [id=0cc2eea0038e4971a3d7119cbaabc735]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.640295 29986 tablet_service.cc:1467] Processing CreateTablet for tablet ed32c167c0d74ee588b59f240d2e5b24 (DEFAULT_TABLE table=rebalance_test_table_16 [id=0cc2eea0038e4971a3d7119cbaabc735]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.640424 30246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4de7cb0be278476481d01083add5c978. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.640643 30245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3f04e7716c54d6090a00d7a6b771b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.640653 29986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed32c167c0d74ee588b59f240d2e5b24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.641366 29985 tablet_service.cc:1467] Processing CreateTablet for tablet c3f04e7716c54d6090a00d7a6b771b64 (DEFAULT_TABLE table=rebalance_test_table_16 [id=0cc2eea0038e4971a3d7119cbaabc735]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.641701 29985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3f04e7716c54d6090a00d7a6b771b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.646203 30114 tablet_service.cc:1467] Processing CreateTablet for tablet c3f04e7716c54d6090a00d7a6b771b64 (DEFAULT_TABLE table=rebalance_test_table_16 [id=0cc2eea0038e4971a3d7119cbaabc735]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:46.646664 30114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3f04e7716c54d6090a00d7a6b771b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:46.668226 32226 tablet_bootstrap.cc:492] T ed32c167c0d74ee588b59f240d2e5b24 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:46.669374 32228 tablet_bootstrap.cc:492] T 4de7cb0be278476481d01083add5c978 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:46.671118 32226 tablet_bootstrap.cc:654] T ed32c167c0d74ee588b59f240d2e5b24 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:46.671850 32228 tablet_bootstrap.cc:654] T 4de7cb0be278476481d01083add5c978 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:46.673633 32225 tablet_bootstrap.cc:492] T c3f04e7716c54d6090a00d7a6b771b64 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:46.675139 32225 tablet_bootstrap.cc:654] T c3f04e7716c54d6090a00d7a6b771b64 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:46.677011 32228 tablet_bootstrap.cc:492] T 4de7cb0be278476481d01083add5c978 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:46.677237 32228 ts_tablet_manager.cc:1367] T 4de7cb0be278476481d01083add5c978 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20240503 02:12:46.677908 32228 raft_consensus.cc:348] T 4de7cb0be278476481d01083add5c978 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:46.678112 32228 raft_consensus.cc:374] T 4de7cb0be278476481d01083add5c978 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:46.678220 32228 raft_consensus.cc:729] T 4de7cb0be278476481d01083add5c978 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:46.678498 32228 consensus_queue.cc:260] T 4de7cb0be278476481d01083add5c978 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:46.679131 32227 tablet_bootstrap.cc:492] T ed32c167c0d74ee588b59f240d2e5b24 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:46.679507 32228 ts_tablet_manager.cc:1398] T 4de7cb0be278476481d01083add5c978 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:12:46.679805 32228 tablet_bootstrap.cc:492] T ed32c167c0d74ee588b59f240d2e5b24 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:46.681473 32228 tablet_bootstrap.cc:654] T ed32c167c0d74ee588b59f240d2e5b24 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:46.681380 32227 tablet_bootstrap.cc:654] T ed32c167c0d74ee588b59f240d2e5b24 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:46.683066 32226 tablet_bootstrap.cc:492] T ed32c167c0d74ee588b59f240d2e5b24 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:46.683276 32226 ts_tablet_manager.cc:1367] T ed32c167c0d74ee588b59f240d2e5b24 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.003s
I20240503 02:12:46.684284 32225 tablet_bootstrap.cc:492] T c3f04e7716c54d6090a00d7a6b771b64 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:46.684185 32226 raft_consensus.cc:348] T ed32c167c0d74ee588b59f240d2e5b24 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:46.684396 32226 raft_consensus.cc:374] T ed32c167c0d74ee588b59f240d2e5b24 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:46.684489 32226 raft_consensus.cc:729] T ed32c167c0d74ee588b59f240d2e5b24 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:46.684639 32225 ts_tablet_manager.cc:1367] T c3f04e7716c54d6090a00d7a6b771b64 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240503 02:12:46.684760 32226 consensus_queue.cc:260] T ed32c167c0d74ee588b59f240d2e5b24 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:46.685511 32226 ts_tablet_manager.cc:1398] T ed32c167c0d74ee588b59f240d2e5b24 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:46.685532 32225 raft_consensus.cc:348] T c3f04e7716c54d6090a00d7a6b771b64 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:46.685802 32225 raft_consensus.cc:374] T c3f04e7716c54d6090a00d7a6b771b64 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:46.685860 32226 tablet_bootstrap.cc:492] T c3f04e7716c54d6090a00d7a6b771b64 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:46.685976 32225 raft_consensus.cc:729] T c3f04e7716c54d6090a00d7a6b771b64 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:46.686261 32225 consensus_queue.cc:260] T c3f04e7716c54d6090a00d7a6b771b64 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:46.688115 32225 ts_tablet_manager.cc:1398] T c3f04e7716c54d6090a00d7a6b771b64 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:46.688454 32225 tablet_bootstrap.cc:492] T 4de7cb0be278476481d01083add5c978 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:46.689185 32226 tablet_bootstrap.cc:654] T c3f04e7716c54d6090a00d7a6b771b64 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:46.690272 32225 tablet_bootstrap.cc:654] T 4de7cb0be278476481d01083add5c978 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:46.695619 32227 tablet_bootstrap.cc:492] T ed32c167c0d74ee588b59f240d2e5b24 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:46.695858 32227 ts_tablet_manager.cc:1367] T ed32c167c0d74ee588b59f240d2e5b24 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20240503 02:12:46.696099 32225 tablet_bootstrap.cc:492] T 4de7cb0be278476481d01083add5c978 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:46.696285 32228 tablet_bootstrap.cc:492] T ed32c167c0d74ee588b59f240d2e5b24 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:46.696306 32225 ts_tablet_manager.cc:1367] T 4de7cb0be278476481d01083add5c978 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240503 02:12:46.696532 32228 ts_tablet_manager.cc:1367] T ed32c167c0d74ee588b59f240d2e5b24 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.006s
I20240503 02:12:46.696542 32226 tablet_bootstrap.cc:492] T c3f04e7716c54d6090a00d7a6b771b64 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:46.696695 32226 ts_tablet_manager.cc:1367] T c3f04e7716c54d6090a00d7a6b771b64 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240503 02:12:46.696792 32227 raft_consensus.cc:348] T ed32c167c0d74ee588b59f240d2e5b24 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:46.697116 32227 raft_consensus.cc:374] T ed32c167c0d74ee588b59f240d2e5b24 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:46.697114 32225 raft_consensus.cc:348] T 4de7cb0be278476481d01083add5c978 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:46.697260 32227 raft_consensus.cc:729] T ed32c167c0d74ee588b59f240d2e5b24 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:46.697378 32225 raft_consensus.cc:374] T 4de7cb0be278476481d01083add5c978 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:46.697379 32228 raft_consensus.cc:348] T ed32c167c0d74ee588b59f240d2e5b24 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:46.697561 32225 raft_consensus.cc:729] T 4de7cb0be278476481d01083add5c978 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:46.697567 32226 raft_consensus.cc:348] T c3f04e7716c54d6090a00d7a6b771b64 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:46.697691 32228 raft_consensus.cc:374] T ed32c167c0d74ee588b59f240d2e5b24 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:46.697618 32227 consensus_queue.cc:260] T ed32c167c0d74ee588b59f240d2e5b24 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:46.697778 32228 raft_consensus.cc:729] T ed32c167c0d74ee588b59f240d2e5b24 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:46.697866 32226 raft_consensus.cc:374] T c3f04e7716c54d6090a00d7a6b771b64 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:46.697979 32226 raft_consensus.cc:729] T c3f04e7716c54d6090a00d7a6b771b64 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:46.698046 32228 consensus_queue.cc:260] T ed32c167c0d74ee588b59f240d2e5b24 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:46.697888 32225 consensus_queue.cc:260] T 4de7cb0be278476481d01083add5c978 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:46.698614 32225 ts_tablet_manager.cc:1398] T 4de7cb0be278476481d01083add5c978 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:46.698748 32228 ts_tablet_manager.cc:1398] T ed32c167c0d74ee588b59f240d2e5b24 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:46.699456 32227 ts_tablet_manager.cc:1398] T ed32c167c0d74ee588b59f240d2e5b24 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:46.699752 32227 tablet_bootstrap.cc:492] T c3f04e7716c54d6090a00d7a6b771b64 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:46.698271 32226 consensus_queue.cc:260] T c3f04e7716c54d6090a00d7a6b771b64 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:46.701602 32227 tablet_bootstrap.cc:654] T c3f04e7716c54d6090a00d7a6b771b64 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:46.703547 32226 ts_tablet_manager.cc:1398] T c3f04e7716c54d6090a00d7a6b771b64 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:12:46.707825 32227 tablet_bootstrap.cc:492] T c3f04e7716c54d6090a00d7a6b771b64 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:46.708003 32227 ts_tablet_manager.cc:1367] T c3f04e7716c54d6090a00d7a6b771b64 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240503 02:12:46.708793 32227 raft_consensus.cc:348] T c3f04e7716c54d6090a00d7a6b771b64 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:46.709053 32227 raft_consensus.cc:374] T c3f04e7716c54d6090a00d7a6b771b64 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:46.709133 32227 raft_consensus.cc:729] T c3f04e7716c54d6090a00d7a6b771b64 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:46.709380 32227 consensus_queue.cc:260] T c3f04e7716c54d6090a00d7a6b771b64 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:46.709815 32227 ts_tablet_manager.cc:1398] T c3f04e7716c54d6090a00d7a6b771b64 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:12:46.710037 32227 tablet_bootstrap.cc:492] T 4de7cb0be278476481d01083add5c978 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:46.711349 32227 tablet_bootstrap.cc:654] T 4de7cb0be278476481d01083add5c978 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:46.715090 32227 tablet_bootstrap.cc:492] T 4de7cb0be278476481d01083add5c978 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:46.715394 32227 ts_tablet_manager.cc:1367] T 4de7cb0be278476481d01083add5c978 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240503 02:12:46.715986 32227 raft_consensus.cc:348] T 4de7cb0be278476481d01083add5c978 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:46.716132 32227 raft_consensus.cc:374] T 4de7cb0be278476481d01083add5c978 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:46.716197 32227 raft_consensus.cc:729] T 4de7cb0be278476481d01083add5c978 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:46.716382 32227 consensus_queue.cc:260] T 4de7cb0be278476481d01083add5c978 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:46.716850 32227 ts_tablet_manager.cc:1398] T 4de7cb0be278476481d01083add5c978 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:12:46.820223 32167 raft_consensus.cc:482] T 4de7cb0be278476481d01083add5c978 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:46.820421 32167 raft_consensus.cc:504] T 4de7cb0be278476481d01083add5c978 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:46.821697 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4de7cb0be278476481d01083add5c978" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:12:46.821779 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4de7cb0be278476481d01083add5c978" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:46.821969 30265 raft_consensus.cc:2454] T 4de7cb0be278476481d01083add5c978 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:12:46.822005 29875 raft_consensus.cc:2454] T 4de7cb0be278476481d01083add5c978 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:12:46.822422 29940 leader_election.cc:302] T 4de7cb0be278476481d01083add5c978 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:46.822733 32137 raft_consensus.cc:2781] T 4de7cb0be278476481d01083add5c978 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:46.822943 32137 raft_consensus.cc:482] T 4de7cb0be278476481d01083add5c978 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:46.822899 32167 leader_election.cc:288] T 4de7cb0be278476481d01083add5c978 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:12:46.823053 32137 raft_consensus.cc:3037] T 4de7cb0be278476481d01083add5c978 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:46.825934 32137 raft_consensus.cc:504] T 4de7cb0be278476481d01083add5c978 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:46.826601 32137 leader_election.cc:288] T 4de7cb0be278476481d01083add5c978 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Requested vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:12:46.827538 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4de7cb0be278476481d01083add5c978" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:12:46.827636 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4de7cb0be278476481d01083add5c978" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:12:46.827729 29875 raft_consensus.cc:3037] T 4de7cb0be278476481d01083add5c978 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:46.827872 30265 raft_consensus.cc:3037] T 4de7cb0be278476481d01083add5c978 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:46.830576 29875 raft_consensus.cc:2454] T 4de7cb0be278476481d01083add5c978 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:12:46.830811 30265 raft_consensus.cc:2454] T 4de7cb0be278476481d01083add5c978 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:12:46.831300 29942 leader_election.cc:302] T 4de7cb0be278476481d01083add5c978 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429; no voters:
I20240503 02:12:46.831607 32137 raft_consensus.cc:2781] T 4de7cb0be278476481d01083add5c978 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:46.831785 32137 raft_consensus.cc:686] T 4de7cb0be278476481d01083add5c978 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Becoming Leader. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Running, Role: LEADER
I20240503 02:12:46.832106 32137 consensus_queue.cc:237] T 4de7cb0be278476481d01083add5c978 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:46.836135 29734 catalog_manager.cc:5484] T 4de7cb0be278476481d01083add5c978 P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: term changed from 0 to 1, leader changed from <none> to 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2). New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:12:46.842713 32137 raft_consensus.cc:482] T ed32c167c0d74ee588b59f240d2e5b24 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:46.842896 32137 raft_consensus.cc:504] T ed32c167c0d74ee588b59f240d2e5b24 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:46.843456 32137 leader_election.cc:288] T ed32c167c0d74ee588b59f240d2e5b24 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:46.843891 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed32c167c0d74ee588b59f240d2e5b24" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:46.843963 30136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed32c167c0d74ee588b59f240d2e5b24" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:12:46.844082 29875 raft_consensus.cc:2454] T ed32c167c0d74ee588b59f240d2e5b24 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:12:46.844230 30136 raft_consensus.cc:2454] T ed32c167c0d74ee588b59f240d2e5b24 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:12:46.844511 29942 leader_election.cc:302] T ed32c167c0d74ee588b59f240d2e5b24 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429; no voters:
I20240503 02:12:46.844812 32137 raft_consensus.cc:2781] T ed32c167c0d74ee588b59f240d2e5b24 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:46.845084 32137 raft_consensus.cc:482] T ed32c167c0d74ee588b59f240d2e5b24 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:46.845216 32137 raft_consensus.cc:3037] T ed32c167c0d74ee588b59f240d2e5b24 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:46.847095 32137 raft_consensus.cc:504] T ed32c167c0d74ee588b59f240d2e5b24 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:46.847630 32137 leader_election.cc:288] T ed32c167c0d74ee588b59f240d2e5b24 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Requested vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:46.848068 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed32c167c0d74ee588b59f240d2e5b24" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:12:46.848083 30136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed32c167c0d74ee588b59f240d2e5b24" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:12:46.848271 30136 raft_consensus.cc:3037] T ed32c167c0d74ee588b59f240d2e5b24 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:46.848259 29875 raft_consensus.cc:3037] T ed32c167c0d74ee588b59f240d2e5b24 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:46.850225 29875 raft_consensus.cc:2454] T ed32c167c0d74ee588b59f240d2e5b24 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:12:46.850260 30136 raft_consensus.cc:2454] T ed32c167c0d74ee588b59f240d2e5b24 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:12:46.850662 29942 leader_election.cc:302] T ed32c167c0d74ee588b59f240d2e5b24 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429; no voters:
I20240503 02:12:46.850934 32137 raft_consensus.cc:2781] T ed32c167c0d74ee588b59f240d2e5b24 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:46.851092 32137 raft_consensus.cc:686] T ed32c167c0d74ee588b59f240d2e5b24 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Becoming Leader. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Running, Role: LEADER
I20240503 02:12:46.851339 32137 consensus_queue.cc:237] T ed32c167c0d74ee588b59f240d2e5b24 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:46.854552 29734 catalog_manager.cc:5484] T ed32c167c0d74ee588b59f240d2e5b24 P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: term changed from 0 to 1, leader changed from <none> to 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2). New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:12:46.937222 32137 raft_consensus.cc:482] T c3f04e7716c54d6090a00d7a6b771b64 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:46.937414 32137 raft_consensus.cc:504] T c3f04e7716c54d6090a00d7a6b771b64 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:46.938022 32137 leader_election.cc:288] T c3f04e7716c54d6090a00d7a6b771b64 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:46.938580 30136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3f04e7716c54d6090a00d7a6b771b64" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:12:46.938684 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3f04e7716c54d6090a00d7a6b771b64" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:12:46.938844 30136 raft_consensus.cc:2454] T c3f04e7716c54d6090a00d7a6b771b64 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:12:46.938886 30265 raft_consensus.cc:2454] T c3f04e7716c54d6090a00d7a6b771b64 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:12:46.939303 29939 leader_election.cc:302] T c3f04e7716c54d6090a00d7a6b771b64 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:46.939616 32137 raft_consensus.cc:2781] T c3f04e7716c54d6090a00d7a6b771b64 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:46.939747 32137 raft_consensus.cc:482] T c3f04e7716c54d6090a00d7a6b771b64 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:46.939842 32137 raft_consensus.cc:3037] T c3f04e7716c54d6090a00d7a6b771b64 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:46.942042 32137 raft_consensus.cc:504] T c3f04e7716c54d6090a00d7a6b771b64 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:46.942585 32137 leader_election.cc:288] T c3f04e7716c54d6090a00d7a6b771b64 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Requested vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:46.943120 30136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3f04e7716c54d6090a00d7a6b771b64" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:12:46.943115 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3f04e7716c54d6090a00d7a6b771b64" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:12:46.943331 30265 raft_consensus.cc:3037] T c3f04e7716c54d6090a00d7a6b771b64 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:46.943336 30136 raft_consensus.cc:3037] T c3f04e7716c54d6090a00d7a6b771b64 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:46.946223 30265 raft_consensus.cc:2454] T c3f04e7716c54d6090a00d7a6b771b64 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:12:46.946235 30136 raft_consensus.cc:2454] T c3f04e7716c54d6090a00d7a6b771b64 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:12:46.946609 29939 leader_election.cc:302] T c3f04e7716c54d6090a00d7a6b771b64 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:46.946904 32137 raft_consensus.cc:2781] T c3f04e7716c54d6090a00d7a6b771b64 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:46.947072 32137 raft_consensus.cc:686] T c3f04e7716c54d6090a00d7a6b771b64 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Becoming Leader. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Running, Role: LEADER
I20240503 02:12:46.947320 32137 consensus_queue.cc:237] T c3f04e7716c54d6090a00d7a6b771b64 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:46.950410 29734 catalog_manager.cc:5484] T c3f04e7716c54d6090a00d7a6b771b64 P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: term changed from 0 to 1, leader changed from <none> to 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2). New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:12:47.340332 32137 consensus_queue.cc:1035] T c3f04e7716c54d6090a00d7a6b771b64 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:47.345469 32137 consensus_queue.cc:1035] T ed32c167c0d74ee588b59f240d2e5b24 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:47.353389 32137 consensus_queue.cc:1035] T ed32c167c0d74ee588b59f240d2e5b24 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:47.393230 32249 consensus_queue.cc:1035] T c3f04e7716c54d6090a00d7a6b771b64 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:47.397442 32253 consensus_queue.cc:1035] T 4de7cb0be278476481d01083add5c978 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:47.415067 32242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:47.415539 32242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 02:12:47.418066 32253 consensus_queue.cc:1035] T 4de7cb0be278476481d01083add5c978 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:47.428571 32242 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_16'
INSERT report
rows total: 2000
time total: 221.436 ms
time per row: 0.110718 ms
I20240503 02:12:47.822852 29734 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalance_test_table_17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:47.843845 30246 tablet_service.cc:1467] Processing CreateTablet for tablet 32a2dbf80daf40d6b21442a39f62f14c (DEFAULT_TABLE table=rebalance_test_table_17 [id=6fdcf94cbd1844948615de6c2a4d18cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:47.843924 29854 tablet_service.cc:1467] Processing CreateTablet for tablet 779a6685fc8c4670a71b51c4493bc2f7 (DEFAULT_TABLE table=rebalance_test_table_17 [id=6fdcf94cbd1844948615de6c2a4d18cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:47.844307 30246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32a2dbf80daf40d6b21442a39f62f14c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:47.844411 29854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 779a6685fc8c4670a71b51c4493bc2f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:47.844791 30245 tablet_service.cc:1467] Processing CreateTablet for tablet 779a6685fc8c4670a71b51c4493bc2f7 (DEFAULT_TABLE table=rebalance_test_table_17 [id=6fdcf94cbd1844948615de6c2a4d18cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:47.844827 29855 tablet_service.cc:1467] Processing CreateTablet for tablet 3f00949b4ca84a49b0f6e62bd7373586 (DEFAULT_TABLE table=rebalance_test_table_17 [id=6fdcf94cbd1844948615de6c2a4d18cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:47.845175 30245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 779a6685fc8c4670a71b51c4493bc2f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:47.845232 29855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f00949b4ca84a49b0f6e62bd7373586. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:47.845844 29853 tablet_service.cc:1467] Processing CreateTablet for tablet 32a2dbf80daf40d6b21442a39f62f14c (DEFAULT_TABLE table=rebalance_test_table_17 [id=6fdcf94cbd1844948615de6c2a4d18cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:47.846216 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32a2dbf80daf40d6b21442a39f62f14c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:47.849499 30115 tablet_service.cc:1467] Processing CreateTablet for tablet 779a6685fc8c4670a71b51c4493bc2f7 (DEFAULT_TABLE table=rebalance_test_table_17 [id=6fdcf94cbd1844948615de6c2a4d18cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:47.849959 30115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 779a6685fc8c4670a71b51c4493bc2f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:47.850117 29986 tablet_service.cc:1467] Processing CreateTablet for tablet 3f00949b4ca84a49b0f6e62bd7373586 (DEFAULT_TABLE table=rebalance_test_table_17 [id=6fdcf94cbd1844948615de6c2a4d18cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:47.850551 29986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f00949b4ca84a49b0f6e62bd7373586. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:47.853457 30114 tablet_service.cc:1467] Processing CreateTablet for tablet 32a2dbf80daf40d6b21442a39f62f14c (DEFAULT_TABLE table=rebalance_test_table_17 [id=6fdcf94cbd1844948615de6c2a4d18cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:47.853873 30114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32a2dbf80daf40d6b21442a39f62f14c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:47.858121 30116 tablet_service.cc:1467] Processing CreateTablet for tablet 3f00949b4ca84a49b0f6e62bd7373586 (DEFAULT_TABLE table=rebalance_test_table_17 [id=6fdcf94cbd1844948615de6c2a4d18cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:47.858666 30116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f00949b4ca84a49b0f6e62bd7373586. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:47.868017 32305 tablet_bootstrap.cc:492] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:47.870190 32305 tablet_bootstrap.cc:654] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.878703 32305 tablet_bootstrap.cc:492] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:47.878994 32305 ts_tablet_manager.cc:1367] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20240503 02:12:47.881845 32305 raft_consensus.cc:348] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:47.882226 32305 raft_consensus.cc:374] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.882377 32305 raft_consensus.cc:729] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.882809 32305 consensus_queue.cc:260] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:47.883466 32305 ts_tablet_manager.cc:1398] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:12:47.898116 32305 tablet_bootstrap.cc:492] T 32a2dbf80daf40d6b21442a39f62f14c P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:47.899912 32305 tablet_bootstrap.cc:654] T 32a2dbf80daf40d6b21442a39f62f14c P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.911335 32307 tablet_bootstrap.cc:492] T 3f00949b4ca84a49b0f6e62bd7373586 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:47.916342 32307 tablet_bootstrap.cc:654] T 3f00949b4ca84a49b0f6e62bd7373586 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.927127 32304 tablet_bootstrap.cc:492] T 779a6685fc8c4670a71b51c4493bc2f7 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:47.929795 32304 tablet_bootstrap.cc:654] T 779a6685fc8c4670a71b51c4493bc2f7 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.937685 32305 tablet_bootstrap.cc:492] T 32a2dbf80daf40d6b21442a39f62f14c P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:47.937899 32307 tablet_bootstrap.cc:492] T 3f00949b4ca84a49b0f6e62bd7373586 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:47.938062 32307 ts_tablet_manager.cc:1367] T 3f00949b4ca84a49b0f6e62bd7373586 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.000s
I20240503 02:12:47.938779 32305 ts_tablet_manager.cc:1367] T 32a2dbf80daf40d6b21442a39f62f14c P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.041s user 0.013s sys 0.023s
I20240503 02:12:47.938951 32307 raft_consensus.cc:348] T 3f00949b4ca84a49b0f6e62bd7373586 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:47.939221 32307 raft_consensus.cc:374] T 3f00949b4ca84a49b0f6e62bd7373586 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.939340 32307 raft_consensus.cc:729] T 3f00949b4ca84a49b0f6e62bd7373586 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.939638 32307 consensus_queue.cc:260] T 3f00949b4ca84a49b0f6e62bd7373586 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:47.940371 32305 raft_consensus.cc:348] T 32a2dbf80daf40d6b21442a39f62f14c P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:47.940730 32305 raft_consensus.cc:374] T 32a2dbf80daf40d6b21442a39f62f14c P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.940834 32305 raft_consensus.cc:729] T 32a2dbf80daf40d6b21442a39f62f14c P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.941198 32305 consensus_queue.cc:260] T 32a2dbf80daf40d6b21442a39f62f14c P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:47.941972 32302 tablet_bootstrap.cc:492] T 779a6685fc8c4670a71b51c4493bc2f7 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:47.943114 32305 ts_tablet_manager.cc:1398] T 32a2dbf80daf40d6b21442a39f62f14c P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:12:47.943435 32305 tablet_bootstrap.cc:492] T 3f00949b4ca84a49b0f6e62bd7373586 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:47.943877 32302 tablet_bootstrap.cc:654] T 779a6685fc8c4670a71b51c4493bc2f7 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.945130 32305 tablet_bootstrap.cc:654] T 3f00949b4ca84a49b0f6e62bd7373586 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.946226 32304 tablet_bootstrap.cc:492] T 779a6685fc8c4670a71b51c4493bc2f7 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:47.946435 32304 ts_tablet_manager.cc:1367] T 779a6685fc8c4670a71b51c4493bc2f7 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.003s
I20240503 02:12:47.947618 32304 raft_consensus.cc:348] T 779a6685fc8c4670a71b51c4493bc2f7 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:47.948005 32304 raft_consensus.cc:374] T 779a6685fc8c4670a71b51c4493bc2f7 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.948166 32304 raft_consensus.cc:729] T 779a6685fc8c4670a71b51c4493bc2f7 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.948534 32304 consensus_queue.cc:260] T 779a6685fc8c4670a71b51c4493bc2f7 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:47.949383 32304 ts_tablet_manager.cc:1398] T 779a6685fc8c4670a71b51c4493bc2f7 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:47.950058 32304 tablet_bootstrap.cc:492] T 32a2dbf80daf40d6b21442a39f62f14c P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:47.947512 32307 ts_tablet_manager.cc:1398] T 3f00949b4ca84a49b0f6e62bd7373586 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 02:12:47.955147 32305 tablet_bootstrap.cc:492] T 3f00949b4ca84a49b0f6e62bd7373586 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:47.955303 32305 ts_tablet_manager.cc:1367] T 3f00949b4ca84a49b0f6e62bd7373586 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20240503 02:12:47.955442 32302 tablet_bootstrap.cc:492] T 779a6685fc8c4670a71b51c4493bc2f7 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:47.955571 32302 ts_tablet_manager.cc:1367] T 779a6685fc8c4670a71b51c4493bc2f7 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.001s
I20240503 02:12:47.956084 32305 raft_consensus.cc:348] T 3f00949b4ca84a49b0f6e62bd7373586 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:47.956295 32305 raft_consensus.cc:374] T 3f00949b4ca84a49b0f6e62bd7373586 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.956293 32302 raft_consensus.cc:348] T 779a6685fc8c4670a71b51c4493bc2f7 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:47.956396 32305 raft_consensus.cc:729] T 3f00949b4ca84a49b0f6e62bd7373586 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.956501 32302 raft_consensus.cc:374] T 779a6685fc8c4670a71b51c4493bc2f7 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.956591 32302 raft_consensus.cc:729] T 779a6685fc8c4670a71b51c4493bc2f7 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.956637 32305 consensus_queue.cc:260] T 3f00949b4ca84a49b0f6e62bd7373586 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:47.956813 32302 consensus_queue.cc:260] T 779a6685fc8c4670a71b51c4493bc2f7 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:47.957324 32302 ts_tablet_manager.cc:1398] T 779a6685fc8c4670a71b51c4493bc2f7 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:12:47.957592 32302 tablet_bootstrap.cc:492] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:47.958248 32305 ts_tablet_manager.cc:1398] T 3f00949b4ca84a49b0f6e62bd7373586 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:12:47.960765 32302 tablet_bootstrap.cc:654] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.969270 32304 tablet_bootstrap.cc:654] T 32a2dbf80daf40d6b21442a39f62f14c P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.971791 32302 tablet_bootstrap.cc:492] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:47.971977 32302 ts_tablet_manager.cc:1367] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.008s
I20240503 02:12:47.973309 32302 raft_consensus.cc:348] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:47.973543 32302 raft_consensus.cc:374] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.973623 32302 raft_consensus.cc:729] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.973879 32304 tablet_bootstrap.cc:492] T 32a2dbf80daf40d6b21442a39f62f14c P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:47.973861 32302 consensus_queue.cc:260] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:47.974062 32304 ts_tablet_manager.cc:1367] T 32a2dbf80daf40d6b21442a39f62f14c P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.003s
I20240503 02:12:47.974403 32302 ts_tablet_manager.cc:1398] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:47.974870 32302 tablet_bootstrap.cc:492] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:47.976159 32302 tablet_bootstrap.cc:654] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:47.976826 32304 raft_consensus.cc:348] T 32a2dbf80daf40d6b21442a39f62f14c P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:47.977121 32304 raft_consensus.cc:374] T 32a2dbf80daf40d6b21442a39f62f14c P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.977208 32304 raft_consensus.cc:729] T 32a2dbf80daf40d6b21442a39f62f14c P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.977466 32304 consensus_queue.cc:260] T 32a2dbf80daf40d6b21442a39f62f14c P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:47.977991 32304 ts_tablet_manager.cc:1398] T 32a2dbf80daf40d6b21442a39f62f14c P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:12:47.978657 32302 tablet_bootstrap.cc:492] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:47.978808 32302 ts_tablet_manager.cc:1367] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240503 02:12:47.979586 32302 raft_consensus.cc:348] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:47.979801 32302 raft_consensus.cc:374] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:47.979889 32302 raft_consensus.cc:729] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:47.980103 32302 consensus_queue.cc:260] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:47.980741 32302 ts_tablet_manager.cc:1398] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:48.071951 32316 raft_consensus.cc:482] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:48.072134 32316 raft_consensus.cc:504] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:48.072898 32316 leader_election.cc:288] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:48.073383 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f00949b4ca84a49b0f6e62bd7373586" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:12:48.073375 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f00949b4ca84a49b0f6e62bd7373586" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:12:48.073616 30004 raft_consensus.cc:2454] T 3f00949b4ca84a49b0f6e62bd7373586 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:12:48.073621 30134 raft_consensus.cc:2454] T 3f00949b4ca84a49b0f6e62bd7373586 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:12:48.074091 29809 leader_election.cc:302] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:48.074472 32316 raft_consensus.cc:2781] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:48.074584 32316 raft_consensus.cc:482] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:48.074687 32316 raft_consensus.cc:3037] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:48.076802 32316 raft_consensus.cc:504] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:48.077414 32316 leader_election.cc:288] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Requested vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:48.078001 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f00949b4ca84a49b0f6e62bd7373586" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:12:48.078047 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f00949b4ca84a49b0f6e62bd7373586" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:12:48.078229 30004 raft_consensus.cc:3037] T 3f00949b4ca84a49b0f6e62bd7373586 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:48.078230 30134 raft_consensus.cc:3037] T 3f00949b4ca84a49b0f6e62bd7373586 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:48.080217 30004 raft_consensus.cc:2454] T 3f00949b4ca84a49b0f6e62bd7373586 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:12:48.080335 30134 raft_consensus.cc:2454] T 3f00949b4ca84a49b0f6e62bd7373586 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:12:48.080628 29809 leader_election.cc:302] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429; no voters:
I20240503 02:12:48.081073 32316 raft_consensus.cc:2781] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:48.081363 32316 raft_consensus.cc:686] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Becoming Leader. State: Replica: 16440e365430405fb10366ab70e2b429, State: Running, Role: LEADER
I20240503 02:12:48.081650 32316 consensus_queue.cc:237] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:48.084968 29733 catalog_manager.cc:5484] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16440e365430405fb10366ab70e2b429 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:12:48.102977 32241 raft_consensus.cc:482] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:48.103171 32241 raft_consensus.cc:504] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:48.104219 32241 leader_election.cc:288] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:12:48.104549 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "779a6685fc8c4670a71b51c4493bc2f7" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:48.104635 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "779a6685fc8c4670a71b51c4493bc2f7" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:12:48.104743 29875 raft_consensus.cc:2454] T 779a6685fc8c4670a71b51c4493bc2f7 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 0.
I20240503 02:12:48.104879 30266 raft_consensus.cc:2454] T 779a6685fc8c4670a71b51c4493bc2f7 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 0.
I20240503 02:12:48.105160 30072 leader_election.cc:302] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:48.105427 32241 raft_consensus.cc:2781] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:48.105571 32241 raft_consensus.cc:482] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:48.105670 32241 raft_consensus.cc:3037] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:48.107659 32241 raft_consensus.cc:504] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:48.108222 32241 leader_election.cc:288] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Requested vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:12:48.108604 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "779a6685fc8c4670a71b51c4493bc2f7" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:12:48.108788 29875 raft_consensus.cc:3037] T 779a6685fc8c4670a71b51c4493bc2f7 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:48.108734 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "779a6685fc8c4670a71b51c4493bc2f7" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:12:48.108947 30266 raft_consensus.cc:3037] T 779a6685fc8c4670a71b51c4493bc2f7 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:48.111054 29875 raft_consensus.cc:2454] T 779a6685fc8c4670a71b51c4493bc2f7 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 1.
I20240503 02:12:48.111114 30266 raft_consensus.cc:2454] T 779a6685fc8c4670a71b51c4493bc2f7 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 1.
I20240503 02:12:48.111500 30072 leader_election.cc:302] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:48.111805 32241 raft_consensus.cc:2781] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:48.112015 32241 raft_consensus.cc:686] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Becoming Leader. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Running, Role: LEADER
I20240503 02:12:48.112325 32241 consensus_queue.cc:237] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:48.116019 29733 catalog_manager.cc:5484] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9094240860e43bfb129193f3bfd5be6 (127.22.108.3). New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:12:48.321578 32316 raft_consensus.cc:482] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:48.321755 32316 raft_consensus.cc:504] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:48.322482 32316 leader_election.cc:288] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:48.322886 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a2dbf80daf40d6b21442a39f62f14c" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:12:48.322906 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a2dbf80daf40d6b21442a39f62f14c" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:12:48.323068 30266 raft_consensus.cc:2454] T 32a2dbf80daf40d6b21442a39f62f14c P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:12:48.323105 30134 raft_consensus.cc:2454] T 32a2dbf80daf40d6b21442a39f62f14c P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:12:48.323486 29809 leader_election.cc:302] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:48.323815 32316 raft_consensus.cc:2781] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:48.323920 32316 raft_consensus.cc:482] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:48.324033 32316 raft_consensus.cc:3037] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:48.326159 32316 raft_consensus.cc:504] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:48.326663 32316 leader_election.cc:288] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Requested vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:48.327142 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a2dbf80daf40d6b21442a39f62f14c" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:12:48.327323 30266 raft_consensus.cc:3037] T 32a2dbf80daf40d6b21442a39f62f14c P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:48.329274 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a2dbf80daf40d6b21442a39f62f14c" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:12:48.329347 30266 raft_consensus.cc:2454] T 32a2dbf80daf40d6b21442a39f62f14c P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:12:48.329447 30134 raft_consensus.cc:3037] T 32a2dbf80daf40d6b21442a39f62f14c P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:48.329774 29809 leader_election.cc:302] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:48.330037 32316 raft_consensus.cc:2781] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:48.330201 32316 raft_consensus.cc:686] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Becoming Leader. State: Replica: 16440e365430405fb10366ab70e2b429, State: Running, Role: LEADER
I20240503 02:12:48.330516 32316 consensus_queue.cc:237] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:48.331741 30134 raft_consensus.cc:2454] T 32a2dbf80daf40d6b21442a39f62f14c P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:12:48.333798 29733 catalog_manager.cc:5484] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16440e365430405fb10366ab70e2b429 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:12:48.476400 32316 consensus_queue.cc:1035] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:48.481810 32316 consensus_queue.cc:1035] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:48.669253 32241 consensus_queue.cc:1035] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:48.682492 32329 consensus_queue.cc:1035] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240503 02:12:48.726794 32320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:48.727166 32320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:48.735139 32320 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_17'
I20240503 02:12:48.817966 32316 consensus_queue.cc:1035] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:48.837949 32316 consensus_queue.cc:1035] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 269.338 ms
time per row: 0.134669 ms
I20240503 02:12:49.171492 29727 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalance_test_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:49.180799 29986 tablet_service.cc:1467] Processing CreateTablet for tablet dc27f3be29a9440c8b3eee46f04890d5 (DEFAULT_TABLE table=rebalance_test_table_18 [id=521f00c464174dec80c6db200c35e142]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:49.181321 29986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc27f3be29a9440c8b3eee46f04890d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:49.181279 29854 tablet_service.cc:1467] Processing CreateTablet for tablet dc27f3be29a9440c8b3eee46f04890d5 (DEFAULT_TABLE table=rebalance_test_table_18 [id=521f00c464174dec80c6db200c35e142]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:49.181711 29854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc27f3be29a9440c8b3eee46f04890d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:49.182711 29984 tablet_service.cc:1467] Processing CreateTablet for tablet 0ae1063008d44f6eaa61a6e353732e48 (DEFAULT_TABLE table=rebalance_test_table_18 [id=521f00c464174dec80c6db200c35e142]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:49.182747 29985 tablet_service.cc:1467] Processing CreateTablet for tablet 8fefbfb003914bf28c4176cd54a9130f (DEFAULT_TABLE table=rebalance_test_table_18 [id=521f00c464174dec80c6db200c35e142]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:49.183143 29984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ae1063008d44f6eaa61a6e353732e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:49.183362 29985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fefbfb003914bf28c4176cd54a9130f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:49.184295 30116 tablet_service.cc:1467] Processing CreateTablet for tablet 0ae1063008d44f6eaa61a6e353732e48 (DEFAULT_TABLE table=rebalance_test_table_18 [id=521f00c464174dec80c6db200c35e142]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:49.184752 30116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ae1063008d44f6eaa61a6e353732e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:49.185174 30245 tablet_service.cc:1467] Processing CreateTablet for tablet dc27f3be29a9440c8b3eee46f04890d5 (DEFAULT_TABLE table=rebalance_test_table_18 [id=521f00c464174dec80c6db200c35e142]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:49.185596 30245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc27f3be29a9440c8b3eee46f04890d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:49.188750 29853 tablet_service.cc:1467] Processing CreateTablet for tablet 8fefbfb003914bf28c4176cd54a9130f (DEFAULT_TABLE table=rebalance_test_table_18 [id=521f00c464174dec80c6db200c35e142]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:49.188890 30246 tablet_service.cc:1467] Processing CreateTablet for tablet 0ae1063008d44f6eaa61a6e353732e48 (DEFAULT_TABLE table=rebalance_test_table_18 [id=521f00c464174dec80c6db200c35e142]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:49.189291 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fefbfb003914bf28c4176cd54a9130f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:49.189291 30246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ae1063008d44f6eaa61a6e353732e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:49.190963 30244 tablet_service.cc:1467] Processing CreateTablet for tablet 8fefbfb003914bf28c4176cd54a9130f (DEFAULT_TABLE table=rebalance_test_table_18 [id=521f00c464174dec80c6db200c35e142]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:49.191350 30244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fefbfb003914bf28c4176cd54a9130f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:49.203474 32384 tablet_bootstrap.cc:492] T 0ae1063008d44f6eaa61a6e353732e48 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:49.206372 32385 tablet_bootstrap.cc:492] T 0ae1063008d44f6eaa61a6e353732e48 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:49.207736 32384 tablet_bootstrap.cc:654] T 0ae1063008d44f6eaa61a6e353732e48 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.209318 32385 tablet_bootstrap.cc:654] T 0ae1063008d44f6eaa61a6e353732e48 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.211009 32383 tablet_bootstrap.cc:492] T dc27f3be29a9440c8b3eee46f04890d5 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:49.212520 32384 tablet_bootstrap.cc:492] T 0ae1063008d44f6eaa61a6e353732e48 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:49.212683 32384 ts_tablet_manager.cc:1367] T 0ae1063008d44f6eaa61a6e353732e48 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240503 02:12:49.212914 32383 tablet_bootstrap.cc:654] T dc27f3be29a9440c8b3eee46f04890d5 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.213599 32384 raft_consensus.cc:348] T 0ae1063008d44f6eaa61a6e353732e48 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:49.213950 32384 raft_consensus.cc:374] T 0ae1063008d44f6eaa61a6e353732e48 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.214111 32384 raft_consensus.cc:729] T 0ae1063008d44f6eaa61a6e353732e48 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.214366 32384 consensus_queue.cc:260] T 0ae1063008d44f6eaa61a6e353732e48 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:49.214952 32385 tablet_bootstrap.cc:492] T 0ae1063008d44f6eaa61a6e353732e48 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:49.215113 32385 ts_tablet_manager.cc:1367] T 0ae1063008d44f6eaa61a6e353732e48 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240503 02:12:49.215970 32385 raft_consensus.cc:348] T 0ae1063008d44f6eaa61a6e353732e48 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:49.216238 32385 raft_consensus.cc:374] T 0ae1063008d44f6eaa61a6e353732e48 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.216380 32384 ts_tablet_manager.cc:1398] T 0ae1063008d44f6eaa61a6e353732e48 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:12:49.216377 32385 raft_consensus.cc:729] T 0ae1063008d44f6eaa61a6e353732e48 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.216697 32385 consensus_queue.cc:260] T 0ae1063008d44f6eaa61a6e353732e48 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:49.216933 32388 tablet_bootstrap.cc:492] T 0ae1063008d44f6eaa61a6e353732e48 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:49.217331 32385 ts_tablet_manager.cc:1398] T 0ae1063008d44f6eaa61a6e353732e48 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:49.217628 32385 tablet_bootstrap.cc:492] T 8fefbfb003914bf28c4176cd54a9130f P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:49.219288 32385 tablet_bootstrap.cc:654] T 8fefbfb003914bf28c4176cd54a9130f P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.221655 32385 tablet_bootstrap.cc:492] T 8fefbfb003914bf28c4176cd54a9130f P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:49.221961 32385 ts_tablet_manager.cc:1367] T 8fefbfb003914bf28c4176cd54a9130f P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:12:49.222749 32383 tablet_bootstrap.cc:492] T dc27f3be29a9440c8b3eee46f04890d5 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:49.222801 32388 tablet_bootstrap.cc:654] T 0ae1063008d44f6eaa61a6e353732e48 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.222913 32383 ts_tablet_manager.cc:1367] T dc27f3be29a9440c8b3eee46f04890d5 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240503 02:12:49.223886 32383 raft_consensus.cc:348] T dc27f3be29a9440c8b3eee46f04890d5 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:49.224109 32383 raft_consensus.cc:374] T dc27f3be29a9440c8b3eee46f04890d5 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.224201 32383 raft_consensus.cc:729] T dc27f3be29a9440c8b3eee46f04890d5 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.224478 32383 consensus_queue.cc:260] T dc27f3be29a9440c8b3eee46f04890d5 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:49.225083 32388 tablet_bootstrap.cc:492] T 0ae1063008d44f6eaa61a6e353732e48 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:49.225174 32383 ts_tablet_manager.cc:1398] T dc27f3be29a9440c8b3eee46f04890d5 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:49.225466 32388 ts_tablet_manager.cc:1367] T 0ae1063008d44f6eaa61a6e353732e48 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.003s
I20240503 02:12:49.226001 32385 raft_consensus.cc:348] T 8fefbfb003914bf28c4176cd54a9130f P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:49.226243 32385 raft_consensus.cc:374] T 8fefbfb003914bf28c4176cd54a9130f P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.226331 32385 raft_consensus.cc:729] T 8fefbfb003914bf28c4176cd54a9130f P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.226411 32388 raft_consensus.cc:348] T 0ae1063008d44f6eaa61a6e353732e48 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:49.226605 32388 raft_consensus.cc:374] T 0ae1063008d44f6eaa61a6e353732e48 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.226704 32388 raft_consensus.cc:729] T 0ae1063008d44f6eaa61a6e353732e48 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.226616 32385 consensus_queue.cc:260] T 8fefbfb003914bf28c4176cd54a9130f P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:49.226959 32388 consensus_queue.cc:260] T 0ae1063008d44f6eaa61a6e353732e48 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:49.227150 32385 ts_tablet_manager.cc:1398] T 8fefbfb003914bf28c4176cd54a9130f P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.005s user 0.000s sys 0.000s
I20240503 02:12:49.227425 32385 tablet_bootstrap.cc:492] T dc27f3be29a9440c8b3eee46f04890d5 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:49.229007 32385 tablet_bootstrap.cc:654] T dc27f3be29a9440c8b3eee46f04890d5 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.229682 32388 ts_tablet_manager.cc:1398] T 0ae1063008d44f6eaa61a6e353732e48 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 02:12:49.229975 32388 tablet_bootstrap.cc:492] T 8fefbfb003914bf28c4176cd54a9130f P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:49.231312 32383 tablet_bootstrap.cc:492] T 8fefbfb003914bf28c4176cd54a9130f P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:49.231696 32388 tablet_bootstrap.cc:654] T 8fefbfb003914bf28c4176cd54a9130f P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.235462 32383 tablet_bootstrap.cc:654] T 8fefbfb003914bf28c4176cd54a9130f P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.237339 32385 tablet_bootstrap.cc:492] T dc27f3be29a9440c8b3eee46f04890d5 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:49.237462 32383 tablet_bootstrap.cc:492] T 8fefbfb003914bf28c4176cd54a9130f P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:49.237490 32385 ts_tablet_manager.cc:1367] T dc27f3be29a9440c8b3eee46f04890d5 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.003s
I20240503 02:12:49.237639 32383 ts_tablet_manager.cc:1367] T 8fefbfb003914bf28c4176cd54a9130f P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 02:12:49.238153 32385 raft_consensus.cc:348] T dc27f3be29a9440c8b3eee46f04890d5 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:49.238317 32385 raft_consensus.cc:374] T dc27f3be29a9440c8b3eee46f04890d5 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.238409 32385 raft_consensus.cc:729] T dc27f3be29a9440c8b3eee46f04890d5 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.238444 32383 raft_consensus.cc:348] T 8fefbfb003914bf28c4176cd54a9130f P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:49.238718 32383 raft_consensus.cc:374] T 8fefbfb003914bf28c4176cd54a9130f P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.238643 32385 consensus_queue.cc:260] T dc27f3be29a9440c8b3eee46f04890d5 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:49.238812 32383 raft_consensus.cc:729] T 8fefbfb003914bf28c4176cd54a9130f P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.239065 32383 consensus_queue.cc:260] T 8fefbfb003914bf28c4176cd54a9130f P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:49.239312 32385 ts_tablet_manager.cc:1398] T dc27f3be29a9440c8b3eee46f04890d5 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:49.239598 32383 ts_tablet_manager.cc:1398] T 8fefbfb003914bf28c4176cd54a9130f P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:49.241328 32388 tablet_bootstrap.cc:492] T 8fefbfb003914bf28c4176cd54a9130f P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:49.241497 32388 ts_tablet_manager.cc:1367] T 8fefbfb003914bf28c4176cd54a9130f P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.002s
I20240503 02:12:49.242178 32388 raft_consensus.cc:348] T 8fefbfb003914bf28c4176cd54a9130f P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:49.242381 32388 raft_consensus.cc:374] T 8fefbfb003914bf28c4176cd54a9130f P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.242455 32388 raft_consensus.cc:729] T 8fefbfb003914bf28c4176cd54a9130f P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.242677 32388 consensus_queue.cc:260] T 8fefbfb003914bf28c4176cd54a9130f P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:49.244206 32388 ts_tablet_manager.cc:1398] T 8fefbfb003914bf28c4176cd54a9130f P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:49.244506 32388 tablet_bootstrap.cc:492] T dc27f3be29a9440c8b3eee46f04890d5 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:49.246284 32388 tablet_bootstrap.cc:654] T dc27f3be29a9440c8b3eee46f04890d5 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.248180 32388 tablet_bootstrap.cc:492] T dc27f3be29a9440c8b3eee46f04890d5 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:49.248334 32388 ts_tablet_manager.cc:1367] T dc27f3be29a9440c8b3eee46f04890d5 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:12:49.249166 32388 raft_consensus.cc:348] T dc27f3be29a9440c8b3eee46f04890d5 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:49.249373 32388 raft_consensus.cc:374] T dc27f3be29a9440c8b3eee46f04890d5 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.249475 32388 raft_consensus.cc:729] T dc27f3be29a9440c8b3eee46f04890d5 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.249728 32388 consensus_queue.cc:260] T dc27f3be29a9440c8b3eee46f04890d5 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:49.250216 32388 ts_tablet_manager.cc:1398] T dc27f3be29a9440c8b3eee46f04890d5 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:49.282248 32379 raft_consensus.cc:482] T 8fefbfb003914bf28c4176cd54a9130f P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:49.282455 32379 raft_consensus.cc:504] T 8fefbfb003914bf28c4176cd54a9130f P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:49.283108 32379 leader_election.cc:288] T 8fefbfb003914bf28c4176cd54a9130f P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:12:49.283686 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fefbfb003914bf28c4176cd54a9130f" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:12:49.283716 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fefbfb003914bf28c4176cd54a9130f" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:49.283933 30265 raft_consensus.cc:2454] T 8fefbfb003914bf28c4176cd54a9130f P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:12:49.284016 29875 raft_consensus.cc:2454] T 8fefbfb003914bf28c4176cd54a9130f P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:12:49.284416 29940 leader_election.cc:302] T 8fefbfb003914bf28c4176cd54a9130f P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:49.284708 32379 raft_consensus.cc:2781] T 8fefbfb003914bf28c4176cd54a9130f P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:49.284839 32379 raft_consensus.cc:482] T 8fefbfb003914bf28c4176cd54a9130f P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:49.284938 32379 raft_consensus.cc:3037] T 8fefbfb003914bf28c4176cd54a9130f P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:49.286840 32379 raft_consensus.cc:504] T 8fefbfb003914bf28c4176cd54a9130f P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:49.287474 32379 leader_election.cc:288] T 8fefbfb003914bf28c4176cd54a9130f P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Requested vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:12:49.287848 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fefbfb003914bf28c4176cd54a9130f" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:12:49.287998 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fefbfb003914bf28c4176cd54a9130f" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:12:49.288074 29875 raft_consensus.cc:3037] T 8fefbfb003914bf28c4176cd54a9130f P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:49.288239 30265 raft_consensus.cc:3037] T 8fefbfb003914bf28c4176cd54a9130f P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:49.290325 29875 raft_consensus.cc:2454] T 8fefbfb003914bf28c4176cd54a9130f P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:12:49.290424 30265 raft_consensus.cc:2454] T 8fefbfb003914bf28c4176cd54a9130f P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:12:49.290787 29942 leader_election.cc:302] T 8fefbfb003914bf28c4176cd54a9130f P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429; no voters:
I20240503 02:12:49.291121 32379 raft_consensus.cc:2781] T 8fefbfb003914bf28c4176cd54a9130f P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:49.291272 32379 raft_consensus.cc:686] T 8fefbfb003914bf28c4176cd54a9130f P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Becoming Leader. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Running, Role: LEADER
I20240503 02:12:49.291534 32379 consensus_queue.cc:237] T 8fefbfb003914bf28c4176cd54a9130f P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:49.294763 29734 catalog_manager.cc:5484] T 8fefbfb003914bf28c4176cd54a9130f P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: term changed from 0 to 1, leader changed from <none> to 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2). New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:12:49.336796 32379 raft_consensus.cc:482] T dc27f3be29a9440c8b3eee46f04890d5 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:49.336989 32379 raft_consensus.cc:504] T dc27f3be29a9440c8b3eee46f04890d5 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:49.337639 32379 leader_election.cc:288] T dc27f3be29a9440c8b3eee46f04890d5 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:49.338243 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc27f3be29a9440c8b3eee46f04890d5" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:49.338239 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc27f3be29a9440c8b3eee46f04890d5" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:12:49.338452 29875 raft_consensus.cc:2454] T dc27f3be29a9440c8b3eee46f04890d5 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:12:49.338439 30265 raft_consensus.cc:2454] T dc27f3be29a9440c8b3eee46f04890d5 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:12:49.338889 29942 leader_election.cc:302] T dc27f3be29a9440c8b3eee46f04890d5 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429; no voters:
I20240503 02:12:49.339179 32379 raft_consensus.cc:2781] T dc27f3be29a9440c8b3eee46f04890d5 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:49.339298 32379 raft_consensus.cc:482] T dc27f3be29a9440c8b3eee46f04890d5 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:49.339383 32379 raft_consensus.cc:3037] T dc27f3be29a9440c8b3eee46f04890d5 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:49.341286 32379 raft_consensus.cc:504] T dc27f3be29a9440c8b3eee46f04890d5 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:49.341904 32379 leader_election.cc:288] T dc27f3be29a9440c8b3eee46f04890d5 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Requested vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:49.342314 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc27f3be29a9440c8b3eee46f04890d5" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:12:49.342418 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc27f3be29a9440c8b3eee46f04890d5" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:12:49.342535 30265 raft_consensus.cc:3037] T dc27f3be29a9440c8b3eee46f04890d5 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:49.342612 29875 raft_consensus.cc:3037] T dc27f3be29a9440c8b3eee46f04890d5 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:49.345459 29875 raft_consensus.cc:2454] T dc27f3be29a9440c8b3eee46f04890d5 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:12:49.345505 30265 raft_consensus.cc:2454] T dc27f3be29a9440c8b3eee46f04890d5 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:12:49.345909 29940 leader_election.cc:302] T dc27f3be29a9440c8b3eee46f04890d5 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:49.346174 32379 raft_consensus.cc:2781] T dc27f3be29a9440c8b3eee46f04890d5 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:49.346390 32379 raft_consensus.cc:686] T dc27f3be29a9440c8b3eee46f04890d5 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Becoming Leader. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Running, Role: LEADER
I20240503 02:12:49.346647 32379 consensus_queue.cc:237] T dc27f3be29a9440c8b3eee46f04890d5 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:49.350143 29734 catalog_manager.cc:5484] T dc27f3be29a9440c8b3eee46f04890d5 P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: term changed from 0 to 1, leader changed from <none> to 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2). New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:12:49.395419 32303 raft_consensus.cc:482] T 0ae1063008d44f6eaa61a6e353732e48 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:49.395593 32303 raft_consensus.cc:504] T 0ae1063008d44f6eaa61a6e353732e48 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:49.396258 32303 leader_election.cc:288] T 0ae1063008d44f6eaa61a6e353732e48 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:49.396669 30136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ae1063008d44f6eaa61a6e353732e48" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:12:49.396674 30006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ae1063008d44f6eaa61a6e353732e48" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:12:49.396878 30006 raft_consensus.cc:2454] T 0ae1063008d44f6eaa61a6e353732e48 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:49.396899 30136 raft_consensus.cc:2454] T 0ae1063008d44f6eaa61a6e353732e48 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:49.397317 30200 leader_election.cc:302] T 0ae1063008d44f6eaa61a6e353732e48 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:49.397564 32303 raft_consensus.cc:2781] T 0ae1063008d44f6eaa61a6e353732e48 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:49.397670 32303 raft_consensus.cc:482] T 0ae1063008d44f6eaa61a6e353732e48 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:49.397765 32303 raft_consensus.cc:3037] T 0ae1063008d44f6eaa61a6e353732e48 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:49.399685 32303 raft_consensus.cc:504] T 0ae1063008d44f6eaa61a6e353732e48 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:49.400202 32303 leader_election.cc:288] T 0ae1063008d44f6eaa61a6e353732e48 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Requested vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:49.400637 30006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ae1063008d44f6eaa61a6e353732e48" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:12:49.400707 30136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ae1063008d44f6eaa61a6e353732e48" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:12:49.400851 30006 raft_consensus.cc:3037] T 0ae1063008d44f6eaa61a6e353732e48 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:49.400918 30136 raft_consensus.cc:3037] T 0ae1063008d44f6eaa61a6e353732e48 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:49.403790 30136 raft_consensus.cc:2454] T 0ae1063008d44f6eaa61a6e353732e48 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:49.404068 30006 raft_consensus.cc:2454] T 0ae1063008d44f6eaa61a6e353732e48 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:49.404201 30199 leader_election.cc:302] T 0ae1063008d44f6eaa61a6e353732e48 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:49.404507 32158 raft_consensus.cc:2781] T 0ae1063008d44f6eaa61a6e353732e48 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:49.404686 32158 raft_consensus.cc:686] T 0ae1063008d44f6eaa61a6e353732e48 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Becoming Leader. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Running, Role: LEADER
I20240503 02:12:49.404968 32158 consensus_queue.cc:237] T 0ae1063008d44f6eaa61a6e353732e48 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:12:49.409209 29734 catalog_manager.cc:5484] T 0ae1063008d44f6eaa61a6e353732e48 P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4). New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:12:49.729982 32379 consensus_queue.cc:1035] T dc27f3be29a9440c8b3eee46f04890d5 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:49.750937 32403 consensus_queue.cc:1035] T dc27f3be29a9440c8b3eee46f04890d5 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:49.814584 32403 consensus_queue.cc:1035] T 8fefbfb003914bf28c4176cd54a9130f P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:49.816046 32399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:49.816587 32399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:49.828619 32399 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:12:49.837127 32403 consensus_queue.cc:1035] T 8fefbfb003914bf28c4176cd54a9130f P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_18'
I20240503 02:12:49.964813 32158 consensus_queue.cc:1035] T 0ae1063008d44f6eaa61a6e353732e48 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:49.971691 32303 consensus_queue.cc:1035] T 0ae1063008d44f6eaa61a6e353732e48 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 269.939 ms
time per row: 0.13497 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 02:12:51.285671 29727 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalance_test_table_19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:51.295030 29984 tablet_service.cc:1467] Processing CreateTablet for tablet c5cd64614c53487ea1c6cc4a4ca5c379 (DEFAULT_TABLE table=rebalance_test_table_19 [id=3c36e49b52c04307a655de366327db8c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:51.295533 29984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5cd64614c53487ea1c6cc4a4ca5c379. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:51.296185 30116 tablet_service.cc:1467] Processing CreateTablet for tablet c5cd64614c53487ea1c6cc4a4ca5c379 (DEFAULT_TABLE table=rebalance_test_table_19 [id=3c36e49b52c04307a655de366327db8c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:51.296211 29853 tablet_service.cc:1467] Processing CreateTablet for tablet c5cd64614c53487ea1c6cc4a4ca5c379 (DEFAULT_TABLE table=rebalance_test_table_19 [id=3c36e49b52c04307a655de366327db8c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:51.296593 30116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5cd64614c53487ea1c6cc4a4ca5c379. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:51.296667 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5cd64614c53487ea1c6cc4a4ca5c379. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:51.297489 30246 tablet_service.cc:1467] Processing CreateTablet for tablet 08ee37b160d441ddabb640be907d6771 (DEFAULT_TABLE table=rebalance_test_table_19 [id=3c36e49b52c04307a655de366327db8c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:51.297904 30246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08ee37b160d441ddabb640be907d6771. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:51.298717 30114 tablet_service.cc:1467] Processing CreateTablet for tablet 08ee37b160d441ddabb640be907d6771 (DEFAULT_TABLE table=rebalance_test_table_19 [id=3c36e49b52c04307a655de366327db8c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:51.299093 30114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08ee37b160d441ddabb640be907d6771. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:51.299024 29985 tablet_service.cc:1467] Processing CreateTablet for tablet 08ee37b160d441ddabb640be907d6771 (DEFAULT_TABLE table=rebalance_test_table_19 [id=3c36e49b52c04307a655de366327db8c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:51.301445 29985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08ee37b160d441ddabb640be907d6771. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:51.302089 29854 tablet_service.cc:1467] Processing CreateTablet for tablet 5040203bd48549ff9c8f64c71d6bc598 (DEFAULT_TABLE table=rebalance_test_table_19 [id=3c36e49b52c04307a655de366327db8c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:51.302176 30244 tablet_service.cc:1467] Processing CreateTablet for tablet 5040203bd48549ff9c8f64c71d6bc598 (DEFAULT_TABLE table=rebalance_test_table_19 [id=3c36e49b52c04307a655de366327db8c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:51.302453 29854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5040203bd48549ff9c8f64c71d6bc598. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:51.302572 30244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5040203bd48549ff9c8f64c71d6bc598. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:51.308599 30115 tablet_service.cc:1467] Processing CreateTablet for tablet 5040203bd48549ff9c8f64c71d6bc598 (DEFAULT_TABLE table=rebalance_test_table_19 [id=3c36e49b52c04307a655de366327db8c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:51.308990 30115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5040203bd48549ff9c8f64c71d6bc598. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:51.310019 32473 tablet_bootstrap.cc:492] T 08ee37b160d441ddabb640be907d6771 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:51.311654 32473 tablet_bootstrap.cc:654] T 08ee37b160d441ddabb640be907d6771 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:51.313635 32473 tablet_bootstrap.cc:492] T 08ee37b160d441ddabb640be907d6771 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:51.313792 32473 ts_tablet_manager.cc:1367] T 08ee37b160d441ddabb640be907d6771 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:12:51.314601 32473 raft_consensus.cc:348] T 08ee37b160d441ddabb640be907d6771 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:51.314824 32473 raft_consensus.cc:374] T 08ee37b160d441ddabb640be907d6771 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:51.314954 32473 raft_consensus.cc:729] T 08ee37b160d441ddabb640be907d6771 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:51.315192 32473 consensus_queue.cc:260] T 08ee37b160d441ddabb640be907d6771 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:51.315837 32473 ts_tablet_manager.cc:1398] T 08ee37b160d441ddabb640be907d6771 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:51.318635 32475 tablet_bootstrap.cc:492] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:51.320485 32476 tablet_bootstrap.cc:492] T 5040203bd48549ff9c8f64c71d6bc598 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:51.320622 32475 tablet_bootstrap.cc:654] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:51.323736 32476 tablet_bootstrap.cc:654] T 5040203bd48549ff9c8f64c71d6bc598 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:51.332263 32473 tablet_bootstrap.cc:492] T c5cd64614c53487ea1c6cc4a4ca5c379 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:51.334527 32473 tablet_bootstrap.cc:654] T c5cd64614c53487ea1c6cc4a4ca5c379 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:51.335047 32472 tablet_bootstrap.cc:492] T c5cd64614c53487ea1c6cc4a4ca5c379 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:51.337347 32476 tablet_bootstrap.cc:492] T 5040203bd48549ff9c8f64c71d6bc598 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:51.337412 32472 tablet_bootstrap.cc:654] T c5cd64614c53487ea1c6cc4a4ca5c379 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:51.337607 32476 ts_tablet_manager.cc:1367] T 5040203bd48549ff9c8f64c71d6bc598 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.004s
I20240503 02:12:51.338698 32476 raft_consensus.cc:348] T 5040203bd48549ff9c8f64c71d6bc598 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:51.339077 32476 raft_consensus.cc:374] T 5040203bd48549ff9c8f64c71d6bc598 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:51.339241 32476 raft_consensus.cc:729] T 5040203bd48549ff9c8f64c71d6bc598 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:51.339659 32476 consensus_queue.cc:260] T 5040203bd48549ff9c8f64c71d6bc598 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:51.342564 32476 ts_tablet_manager.cc:1398] T 5040203bd48549ff9c8f64c71d6bc598 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 02:12:51.343076 32476 tablet_bootstrap.cc:492] T 08ee37b160d441ddabb640be907d6771 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:51.346657 32476 tablet_bootstrap.cc:654] T 08ee37b160d441ddabb640be907d6771 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:51.349627 32473 tablet_bootstrap.cc:492] T c5cd64614c53487ea1c6cc4a4ca5c379 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:51.349958 32473 ts_tablet_manager.cc:1367] T c5cd64614c53487ea1c6cc4a4ca5c379 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.012s
I20240503 02:12:51.350481 32472 tablet_bootstrap.cc:492] T c5cd64614c53487ea1c6cc4a4ca5c379 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:51.350751 32472 ts_tablet_manager.cc:1367] T c5cd64614c53487ea1c6cc4a4ca5c379 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.004s
I20240503 02:12:51.351110 32473 raft_consensus.cc:348] T c5cd64614c53487ea1c6cc4a4ca5c379 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:51.351502 32473 raft_consensus.cc:374] T c5cd64614c53487ea1c6cc4a4ca5c379 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:51.351549 32472 raft_consensus.cc:348] T c5cd64614c53487ea1c6cc4a4ca5c379 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:51.351644 32473 raft_consensus.cc:729] T c5cd64614c53487ea1c6cc4a4ca5c379 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:51.351795 32472 raft_consensus.cc:374] T c5cd64614c53487ea1c6cc4a4ca5c379 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:51.351907 32472 raft_consensus.cc:729] T c5cd64614c53487ea1c6cc4a4ca5c379 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:51.352074 32473 consensus_queue.cc:260] T c5cd64614c53487ea1c6cc4a4ca5c379 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:51.352151 32472 consensus_queue.cc:260] T c5cd64614c53487ea1c6cc4a4ca5c379 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:51.353719 32472 ts_tablet_manager.cc:1398] T c5cd64614c53487ea1c6cc4a4ca5c379 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:12:51.354398 32472 tablet_bootstrap.cc:492] T 5040203bd48549ff9c8f64c71d6bc598 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:51.341719 32475 tablet_bootstrap.cc:492] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:51.357470 32472 tablet_bootstrap.cc:654] T 5040203bd48549ff9c8f64c71d6bc598 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:51.357563 32475 ts_tablet_manager.cc:1367] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.014s
I20240503 02:12:51.358433 32475 raft_consensus.cc:348] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:51.358763 32475 raft_consensus.cc:374] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:51.352877 32473 ts_tablet_manager.cc:1398] T c5cd64614c53487ea1c6cc4a4ca5c379 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:12:51.359210 32475 raft_consensus.cc:729] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:51.360440 32475 consensus_queue.cc:260] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:51.360826 32476 tablet_bootstrap.cc:492] T 08ee37b160d441ddabb640be907d6771 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:51.361053 32476 ts_tablet_manager.cc:1367] T 08ee37b160d441ddabb640be907d6771 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.004s
I20240503 02:12:51.361182 32475 ts_tablet_manager.cc:1398] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:51.361793 32476 raft_consensus.cc:348] T 08ee37b160d441ddabb640be907d6771 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:51.362008 32476 raft_consensus.cc:374] T 08ee37b160d441ddabb640be907d6771 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:51.362089 32476 raft_consensus.cc:729] T 08ee37b160d441ddabb640be907d6771 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:51.362310 32475 tablet_bootstrap.cc:492] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:51.362390 32453 raft_consensus.cc:482] T 08ee37b160d441ddabb640be907d6771 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:51.362326 32476 consensus_queue.cc:260] T 08ee37b160d441ddabb640be907d6771 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:51.362563 32453 raft_consensus.cc:504] T 08ee37b160d441ddabb640be907d6771 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:51.363216 32453 leader_election.cc:288] T 08ee37b160d441ddabb640be907d6771 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:51.363327 32476 ts_tablet_manager.cc:1398] T 08ee37b160d441ddabb640be907d6771 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:12:51.364270 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08ee37b160d441ddabb640be907d6771" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:12:51.364516 30265 raft_consensus.cc:2454] T 08ee37b160d441ddabb640be907d6771 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:12:51.364616 32472 tablet_bootstrap.cc:492] T 5040203bd48549ff9c8f64c71d6bc598 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:51.364850 32472 ts_tablet_manager.cc:1367] T 5040203bd48549ff9c8f64c71d6bc598 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20240503 02:12:51.365125 29940 leader_election.cc:302] T 08ee37b160d441ddabb640be907d6771 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:12:51.365792 32453 raft_consensus.cc:2781] T 08ee37b160d441ddabb640be907d6771 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:51.365933 32453 raft_consensus.cc:482] T 08ee37b160d441ddabb640be907d6771 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:51.365842 32472 raft_consensus.cc:348] T 5040203bd48549ff9c8f64c71d6bc598 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:51.366027 32453 raft_consensus.cc:3037] T 08ee37b160d441ddabb640be907d6771 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:51.366035 32472 raft_consensus.cc:374] T 5040203bd48549ff9c8f64c71d6bc598 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:51.366101 32472 raft_consensus.cc:729] T 5040203bd48549ff9c8f64c71d6bc598 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:51.366473 32475 tablet_bootstrap.cc:654] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:51.366328 32472 consensus_queue.cc:260] T 5040203bd48549ff9c8f64c71d6bc598 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:51.366672 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08ee37b160d441ddabb640be907d6771" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:12:51.368170 32472 ts_tablet_manager.cc:1398] T 5040203bd48549ff9c8f64c71d6bc598 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20240503 02:12:51.368505 29939 leader_election.cc:341] T 08ee37b160d441ddabb640be907d6771 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:51.368664 32453 raft_consensus.cc:504] T 08ee37b160d441ddabb640be907d6771 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:51.371575 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08ee37b160d441ddabb640be907d6771" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:12:51.371793 30265 raft_consensus.cc:3037] T 08ee37b160d441ddabb640be907d6771 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:51.373310 32453 leader_election.cc:288] T 08ee37b160d441ddabb640be907d6771 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Requested vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:12:51.374022 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08ee37b160d441ddabb640be907d6771" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
W20240503 02:12:51.374581 29939 leader_election.cc:341] T 08ee37b160d441ddabb640be907d6771 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:51.374614 30265 raft_consensus.cc:2454] T 08ee37b160d441ddabb640be907d6771 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:12:51.374992 29940 leader_election.cc:302] T 08ee37b160d441ddabb640be907d6771 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 1f10f80f13c84e80b03eda7085a3b012; no voters: a9094240860e43bfb129193f3bfd5be6
I20240503 02:12:51.375371 32453 raft_consensus.cc:2781] T 08ee37b160d441ddabb640be907d6771 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:51.375514 32453 raft_consensus.cc:686] T 08ee37b160d441ddabb640be907d6771 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Becoming Leader. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Running, Role: LEADER
I20240503 02:12:51.375720 32475 tablet_bootstrap.cc:492] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:51.375864 32475 ts_tablet_manager.cc:1367] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.001s
I20240503 02:12:51.375793 32453 consensus_queue.cc:237] T 08ee37b160d441ddabb640be907d6771 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:51.376649 32475 raft_consensus.cc:348] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:51.378396 32475 raft_consensus.cc:374] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:51.378533 32475 raft_consensus.cc:729] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:51.379019 32475 consensus_queue.cc:260] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:51.379642 32475 ts_tablet_manager.cc:1398] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:12:51.379703 29727 catalog_manager.cc:5484] T 08ee37b160d441ddabb640be907d6771 P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: term changed from 0 to 1, leader changed from <none> to 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2). New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:12:51.380030 32475 tablet_bootstrap.cc:492] T 08ee37b160d441ddabb640be907d6771 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:51.381765 32475 tablet_bootstrap.cc:654] T 08ee37b160d441ddabb640be907d6771 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:51.384779 32475 tablet_bootstrap.cc:492] T 08ee37b160d441ddabb640be907d6771 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:51.384970 32475 ts_tablet_manager.cc:1367] T 08ee37b160d441ddabb640be907d6771 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240503 02:12:51.385872 32475 raft_consensus.cc:348] T 08ee37b160d441ddabb640be907d6771 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:51.386044 32475 raft_consensus.cc:374] T 08ee37b160d441ddabb640be907d6771 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:51.386143 32475 raft_consensus.cc:729] T 08ee37b160d441ddabb640be907d6771 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:51.386413 32475 consensus_queue.cc:260] T 08ee37b160d441ddabb640be907d6771 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:51.387014 32475 ts_tablet_manager.cc:1398] T 08ee37b160d441ddabb640be907d6771 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:51.453531 32330 raft_consensus.cc:482] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:51.453753 32330 raft_consensus.cc:504] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:51.454516 32330 leader_election.cc:288] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:12:51.454821 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5040203bd48549ff9c8f64c71d6bc598" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:51.455085 29875 raft_consensus.cc:2454] T 5040203bd48549ff9c8f64c71d6bc598 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 0.
I20240503 02:12:51.455195 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5040203bd48549ff9c8f64c71d6bc598" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:12:51.455430 30266 raft_consensus.cc:2454] T 5040203bd48549ff9c8f64c71d6bc598 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 0.
I20240503 02:12:51.455683 30072 leader_election.cc:302] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:51.455977 32330 raft_consensus.cc:2781] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:51.456104 32330 raft_consensus.cc:482] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:51.456193 32330 raft_consensus.cc:3037] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:51.458180 32330 raft_consensus.cc:504] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:51.458778 32330 leader_election.cc:288] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Requested vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:12:51.459295 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5040203bd48549ff9c8f64c71d6bc598" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:12:51.459517 30266 raft_consensus.cc:3037] T 5040203bd48549ff9c8f64c71d6bc598 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:51.459712 29874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5040203bd48549ff9c8f64c71d6bc598" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:12:51.459911 29874 raft_consensus.cc:3037] T 5040203bd48549ff9c8f64c71d6bc598 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:51.462440 30266 raft_consensus.cc:2454] T 5040203bd48549ff9c8f64c71d6bc598 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 1.
I20240503 02:12:51.462690 29874 raft_consensus.cc:2454] T 5040203bd48549ff9c8f64c71d6bc598 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 1.
I20240503 02:12:51.462899 30070 leader_election.cc:302] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:51.463486 32486 raft_consensus.cc:2781] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:51.464175 32486 raft_consensus.cc:686] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Becoming Leader. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Running, Role: LEADER
I20240503 02:12:51.464497 32486 consensus_queue.cc:237] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:51.467738 29734 catalog_manager.cc:5484] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9094240860e43bfb129193f3bfd5be6 (127.22.108.3). New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:12:51.509899 32486 raft_consensus.cc:482] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:51.510149 32486 raft_consensus.cc:504] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:51.510867 32486 leader_election.cc:288] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:51.511358 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5cd64614c53487ea1c6cc4a4ca5c379" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:12:51.511562 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5cd64614c53487ea1c6cc4a4ca5c379" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:51.511660 30005 raft_consensus.cc:2454] T c5cd64614c53487ea1c6cc4a4ca5c379 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 0.
I20240503 02:12:51.511811 29875 raft_consensus.cc:2454] T c5cd64614c53487ea1c6cc4a4ca5c379 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 0.
I20240503 02:12:51.512115 30070 leader_election.cc:302] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:51.513134 32486 raft_consensus.cc:2781] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:51.513250 32486 raft_consensus.cc:482] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:51.513391 32486 raft_consensus.cc:3037] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:51.515635 32486 raft_consensus.cc:504] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:51.516213 32486 leader_election.cc:288] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Requested vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:51.516633 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5cd64614c53487ea1c6cc4a4ca5c379" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:12:51.516855 30005 raft_consensus.cc:3037] T c5cd64614c53487ea1c6cc4a4ca5c379 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:51.516916 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5cd64614c53487ea1c6cc4a4ca5c379" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:12:51.517174 29875 raft_consensus.cc:3037] T c5cd64614c53487ea1c6cc4a4ca5c379 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:51.519176 29875 raft_consensus.cc:2454] T c5cd64614c53487ea1c6cc4a4ca5c379 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 1.
I20240503 02:12:51.519345 30005 raft_consensus.cc:2454] T c5cd64614c53487ea1c6cc4a4ca5c379 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 1.
I20240503 02:12:51.519620 30072 leader_election.cc:302] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:51.519938 32486 raft_consensus.cc:2781] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:51.520088 32486 raft_consensus.cc:686] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Becoming Leader. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Running, Role: LEADER
I20240503 02:12:51.520372 32486 consensus_queue.cc:237] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:51.523803 29734 catalog_manager.cc:5484] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9094240860e43bfb129193f3bfd5be6 (127.22.108.3). New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:12:51.701370 32486 raft_consensus.cc:482] T 08ee37b160d441ddabb640be907d6771 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:51.702010 32486 raft_consensus.cc:504] T 08ee37b160d441ddabb640be907d6771 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:12:51.702876 32486 leader_election.cc:288] T 08ee37b160d441ddabb640be907d6771 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:12:51.703393 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08ee37b160d441ddabb640be907d6771" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:12:51.703466 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08ee37b160d441ddabb640be907d6771" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:12:51.703655 30266 raft_consensus.cc:2379] T 08ee37b160d441ddabb640be907d6771 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a9094240860e43bfb129193f3bfd5be6 in current term 1: Already voted for candidate 155735c7f8b14997ac4c770aee2a9401 in this term.
I20240503 02:12:51.704252 30070 leader_election.cc:302] T 08ee37b160d441ddabb640be907d6771 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6; no voters: 155735c7f8b14997ac4c770aee2a9401, 1f10f80f13c84e80b03eda7085a3b012
I20240503 02:12:51.704663 32486 raft_consensus.cc:3037] T 08ee37b160d441ddabb640be907d6771 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:51.707059 32486 raft_consensus.cc:2726] T 08ee37b160d441ddabb640be907d6771 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:51.929538 32490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:51.929853 32490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:51.937651 32490 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:12:51.943145 32453 consensus_queue.cc:1035] T 08ee37b160d441ddabb640be907d6771 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:51.958307 32453 consensus_queue.cc:1035] T 08ee37b160d441ddabb640be907d6771 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:51.958751 32486 consensus_queue.cc:1035] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:51.967339 32487 consensus_queue.cc:1035] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_19'
I20240503 02:12:52.027005 32486 consensus_queue.cc:1035] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:52.034421 32487 consensus_queue.cc:1035] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 241.733 ms
time per row: 0.120866 ms
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240503 02:12:53.346518 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.5:35595
--local_ip_for_outbound_sockets=127.22.108.5
--tserver_master_addrs=127.22.108.62:40071
--webserver_port=40379
--webserver_interface=127.22.108.5
--builtin_ntp_servers=127.22.108.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 02:12:53.715595 32557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:53.715888 32557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:53.716092 32557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:53.725147 32557 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:12:53.725268 32557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:53.725453 32557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.5
I20240503 02:12:53.735471 32557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.5:35595
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.108.5
--webserver_port=40379
--tserver_master_addrs=127.22.108.62:40071
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.5
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:53.737064 32557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:53.739166 32557 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:53.753481 32564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:53.754141 32563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:53.754735 32566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:53.755741 32557 server_base.cc:1034] running on GCE node
I20240503 02:12:53.756362 32557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:53.757369 32557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:53.758569 32557 hybrid_clock.cc:648] HybridClock initialized: now 1714702373758534 us; error 44 us; skew 500 ppm
I20240503 02:12:53.758904 32557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:53.766968 32557 webserver.cc:460] Webserver started at http://127.22.108.5:40379/ using document root <none> and password file <none>
I20240503 02:12:53.767799 32557 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:53.767928 32557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:53.773732 32557 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20240503 02:12:53.776666 32572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:53.778133 32557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:12:53.778293 32557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "9a4d92bc25374fb3960fde7d10be309d"
format_stamp: "Formatted at 2024-05-03 02:12:19 on dist-test-slave-sc15"
I20240503 02:12:53.779294 32557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:53.794819 32557 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:53.795616 32557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:53.795895 32557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:53.796975 32557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:53.798640 32557 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:53.798718 32557 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:53.798841 32557 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:53.798913 32557 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:53.849761 32557 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.5:35595
I20240503 02:12:53.849815 32686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.5:35595 every 8 connection(s)
I20240503 02:12:53.851229 32557 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:12:53.851881 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 32557
I20240503 02:12:53.852661 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.6:39975
--local_ip_for_outbound_sockets=127.22.108.6
--tserver_master_addrs=127.22.108.62:40071
--webserver_port=35081
--webserver_interface=127.22.108.6
--builtin_ntp_servers=127.22.108.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:12:53.866629 32687 heartbeater.cc:347] Connected to a master server at 127.22.108.62:40071
I20240503 02:12:53.866990 32687 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:53.867950 32687 heartbeater.cc:511] Master 127.22.108.62:40071 requested a full tablet report, sending...
I20240503 02:12:53.869710 29733 ts_manager.cc:194] Re-registered known tserver with Master: 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:12:53.870935 29733 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.5:43111
W20240503 02:12:54.218519 32692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:54.218850 32692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:54.218991 32692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:54.227084 32692 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:12:54.227241 32692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:54.227403 32692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.6
I20240503 02:12:54.237274 32692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.6:39975
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.22.108.6
--webserver_port=35081
--tserver_master_addrs=127.22.108.62:40071
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.6
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:54.238781 32692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:54.240917 32692 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:54.254524 32697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:54.255373 32692 server_base.cc:1034] running on GCE node
W20240503 02:12:54.255920 32698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:54.256475 32700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:54.257325 32692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:54.258453 32692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:54.259707 32692 hybrid_clock.cc:648] HybridClock initialized: now 1714702374259655 us; error 45 us; skew 500 ppm
I20240503 02:12:54.260053 32692 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:54.263514 32692 webserver.cc:460] Webserver started at http://127.22.108.6:35081/ using document root <none> and password file <none>
I20240503 02:12:54.264333 32692 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:54.264446 32692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:54.270481 32692 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20240503 02:12:54.273393 32707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:54.274829 32692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240503 02:12:54.275014 32692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "8a666e798fe8477d8dd1e51b385d5ead"
format_stamp: "Formatted at 2024-05-03 02:12:19 on dist-test-slave-sc15"
I20240503 02:12:54.275840 32692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:54.299899 32692 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:54.300784 32692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:54.301107 32692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:54.302268 32692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:54.304004 32692 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:54.304088 32692 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:54.304220 32692 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:54.304277 32692 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:54.356338 32692 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.6:39975
I20240503 02:12:54.356865 351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.6:39975 every 8 connection(s)
I20240503 02:12:54.358136 32692 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 02:12:54.359599 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 32692
I20240503 02:12:54.360599 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.7:44241
--local_ip_for_outbound_sockets=127.22.108.7
--tserver_master_addrs=127.22.108.62:40071
--webserver_port=40935
--webserver_interface=127.22.108.7
--builtin_ntp_servers=127.22.108.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:12:54.385836 352 heartbeater.cc:347] Connected to a master server at 127.22.108.62:40071
I20240503 02:12:54.386204 352 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:54.387208 352 heartbeater.cc:511] Master 127.22.108.62:40071 requested a full tablet report, sending...
I20240503 02:12:54.388777 29733 ts_manager.cc:194] Re-registered known tserver with Master: 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:12:54.390087 29733 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.6:47259
W20240503 02:12:54.735105 355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:54.735484 355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:54.735690 355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:54.744462 355 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:12:54.744586 355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:54.744767 355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.7
I20240503 02:12:54.754714 355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.7:44241
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.22.108.7
--webserver_port=40935
--tserver_master_addrs=127.22.108.62:40071
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.7
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:54.756178 355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:54.758383 355 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:54.771037 360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:54.772650 363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:54.773102 361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:54.773747 355 server_base.cc:1034] running on GCE node
I20240503 02:12:54.774343 355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:54.775456 355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:54.776710 355 hybrid_clock.cc:648] HybridClock initialized: now 1714702374776633 us; error 47 us; skew 500 ppm
I20240503 02:12:54.777069 355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:54.779429 355 webserver.cc:460] Webserver started at http://127.22.108.7:40935/ using document root <none> and password file <none>
I20240503 02:12:54.780221 355 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:54.780351 355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:54.786067 355 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20240503 02:12:54.788712 369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:54.790212 355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240503 02:12:54.790395 355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "fbbc2be6382e47908ecb7c68c367bd26"
format_stamp: "Formatted at 2024-05-03 02:12:20 on dist-test-slave-sc15"
I20240503 02:12:54.791429 355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:54.825959 355 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:54.826992 355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:54.827312 355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:54.828420 355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:54.830226 355 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:54.830300 355 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:54.830392 355 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:54.830435 355 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:54.868043 355 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.7:44241
I20240503 02:12:54.868145 481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.7:44241 every 8 connection(s)
I20240503 02:12:54.869925 355 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 02:12:54.874084 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 355
I20240503 02:12:54.874455 32687 heartbeater.cc:503] Master 127.22.108.62:40071 was elected leader, sending a full tablet report...
I20240503 02:12:54.874861 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.8:42233
--local_ip_for_outbound_sockets=127.22.108.8
--tserver_master_addrs=127.22.108.62:40071
--webserver_port=46641
--webserver_interface=127.22.108.8
--builtin_ntp_servers=127.22.108.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:12:54.895479 482 heartbeater.cc:347] Connected to a master server at 127.22.108.62:40071
I20240503 02:12:54.895852 482 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:54.896894 482 heartbeater.cc:511] Master 127.22.108.62:40071 requested a full tablet report, sending...
I20240503 02:12:54.898700 29733 ts_manager.cc:194] Re-registered known tserver with Master: fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:12:54.899997 29733 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.7:43593
W20240503 02:12:55.255848 485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:55.256212 485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:55.256408 485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:55.264828 485 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:12:55.264956 485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:55.265122 485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.8
I20240503 02:12:55.274624 485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.8:42233
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.22.108.8
--webserver_port=46641
--tserver_master_addrs=127.22.108.62:40071
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.8
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:55.276320 485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:55.278455 485 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:55.291637 491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:55.292107 494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:55.292954 492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:55.293659 485 server_base.cc:1034] running on GCE node
I20240503 02:12:55.294308 485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:55.295367 485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:55.296587 485 hybrid_clock.cc:648] HybridClock initialized: now 1714702375296533 us; error 50 us; skew 500 ppm
I20240503 02:12:55.296929 485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:55.299640 485 webserver.cc:460] Webserver started at http://127.22.108.8:46641/ using document root <none> and password file <none>
I20240503 02:12:55.300622 485 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:55.300758 485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:55.307031 485 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20240503 02:12:55.310492 500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:55.312011 485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:12:55.312213 485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
format_stamp: "Formatted at 2024-05-03 02:12:20 on dist-test-slave-sc15"
I20240503 02:12:55.313109 485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:55.333642 485 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:55.334609 485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:55.334894 485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:55.335932 485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:55.337455 485 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:55.337535 485 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:55.337661 485 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:55.337715 485 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:55.392274 485 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.8:42233
I20240503 02:12:55.392331 612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.8:42233 every 8 connection(s)
I20240503 02:12:55.392875 352 heartbeater.cc:503] Master 127.22.108.62:40071 was elected leader, sending a full tablet report...
I20240503 02:12:55.394019 485 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 02:12:55.395136 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 485
I20240503 02:12:55.398937 29727 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:12:55.411644 613 heartbeater.cc:347] Connected to a master server at 127.22.108.62:40071
I20240503 02:12:55.412010 613 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:55.412957 613 heartbeater.cc:511] Master 127.22.108.62:40071 requested a full tablet report, sending...
I20240503 02:12:55.414654 29733 ts_manager.cc:194] Re-registered known tserver with Master: f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:12:55.415786 29733 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.8:42779
I20240503 02:12:55.434067 29984 tablet_service.cc:1467] Processing CreateTablet for tablet 2b1d89681c624e0dad39685bd090ce86 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=19ebc3eaa8f54e818d1981b70d2b9f58]), partition=
I20240503 02:12:55.434613 29984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1d89681c624e0dad39685bd090ce86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:55.462972 628 tablet_bootstrap.cc:492] T 2b1d89681c624e0dad39685bd090ce86 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:55.464340 417 tablet_service.cc:1467] Processing CreateTablet for tablet 2b1d89681c624e0dad39685bd090ce86 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=19ebc3eaa8f54e818d1981b70d2b9f58]), partition=
I20240503 02:12:55.465777 417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1d89681c624e0dad39685bd090ce86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:55.467707 628 tablet_bootstrap.cc:654] T 2b1d89681c624e0dad39685bd090ce86 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:55.471946 32755 tablet_service.cc:1467] Processing CreateTablet for tablet 2b1d89681c624e0dad39685bd090ce86 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=19ebc3eaa8f54e818d1981b70d2b9f58]), partition=
I20240503 02:12:55.475451 32755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1d89681c624e0dad39685bd090ce86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:55.482049 628 tablet_bootstrap.cc:492] T 2b1d89681c624e0dad39685bd090ce86 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:55.482290 628 ts_tablet_manager.cc:1367] T 2b1d89681c624e0dad39685bd090ce86 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20240503 02:12:55.483103 628 raft_consensus.cc:348] T 2b1d89681c624e0dad39685bd090ce86 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:55.483503 628 raft_consensus.cc:374] T 2b1d89681c624e0dad39685bd090ce86 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:55.483666 628 raft_consensus.cc:729] T 2b1d89681c624e0dad39685bd090ce86 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:55.484050 628 consensus_queue.cc:260] T 2b1d89681c624e0dad39685bd090ce86 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:55.484892 628 ts_tablet_manager.cc:1398] T 2b1d89681c624e0dad39685bd090ce86 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:55.488884 632 tablet_bootstrap.cc:492] T 2b1d89681c624e0dad39685bd090ce86 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:12:55.492555 633 tablet_bootstrap.cc:492] T 2b1d89681c624e0dad39685bd090ce86 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:12:55.495440 633 tablet_bootstrap.cc:654] T 2b1d89681c624e0dad39685bd090ce86 P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:55.495532 632 tablet_bootstrap.cc:654] T 2b1d89681c624e0dad39685bd090ce86 P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:55.496587 633 log.cc:824] T 2b1d89681c624e0dad39685bd090ce86 P 8a666e798fe8477d8dd1e51b385d5ead: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:55.496687 632 log.cc:824] T 2b1d89681c624e0dad39685bd090ce86 P fbbc2be6382e47908ecb7c68c367bd26: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:55.499547 633 tablet_bootstrap.cc:492] T 2b1d89681c624e0dad39685bd090ce86 P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:12:55.499819 633 ts_tablet_manager.cc:1367] T 2b1d89681c624e0dad39685bd090ce86 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20240503 02:12:55.504065 632 tablet_bootstrap.cc:492] T 2b1d89681c624e0dad39685bd090ce86 P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:12:55.504364 632 ts_tablet_manager.cc:1367] T 2b1d89681c624e0dad39685bd090ce86 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.005s
I20240503 02:12:55.509608 633 raft_consensus.cc:348] T 2b1d89681c624e0dad39685bd090ce86 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:55.509982 633 raft_consensus.cc:374] T 2b1d89681c624e0dad39685bd090ce86 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:55.510094 633 raft_consensus.cc:729] T 2b1d89681c624e0dad39685bd090ce86 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:12:55.510757 633 consensus_queue.cc:260] T 2b1d89681c624e0dad39685bd090ce86 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:55.512454 633 ts_tablet_manager.cc:1398] T 2b1d89681c624e0dad39685bd090ce86 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.013s user 0.006s sys 0.005s
I20240503 02:12:55.513618 632 raft_consensus.cc:348] T 2b1d89681c624e0dad39685bd090ce86 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:55.514750 632 raft_consensus.cc:374] T 2b1d89681c624e0dad39685bd090ce86 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:55.514878 632 raft_consensus.cc:729] T 2b1d89681c624e0dad39685bd090ce86 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:12:55.515681 632 consensus_queue.cc:260] T 2b1d89681c624e0dad39685bd090ce86 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:55.516846 482 heartbeater.cc:503] Master 127.22.108.62:40071 was elected leader, sending a full tablet report...
I20240503 02:12:55.517587 632 ts_tablet_manager.cc:1398] T 2b1d89681c624e0dad39685bd090ce86 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.013s user 0.012s sys 0.002s
W20240503 02:12:55.577647 30052 tablet.cc:2369] T 2b1d89681c624e0dad39685bd090ce86 P 155735c7f8b14997ac4c770aee2a9401: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:12:55.622537 483 tablet.cc:2369] T 2b1d89681c624e0dad39685bd090ce86 P fbbc2be6382e47908ecb7c68c367bd26: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:12:55.623926 353 tablet.cc:2369] T 2b1d89681c624e0dad39685bd090ce86 P 8a666e798fe8477d8dd1e51b385d5ead: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:12:55.727548 636 raft_consensus.cc:482] T 2b1d89681c624e0dad39685bd090ce86 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:55.727845 636 raft_consensus.cc:504] T 2b1d89681c624e0dad39685bd090ce86 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:55.730255 636 leader_election.cc:288] T 2b1d89681c624e0dad39685bd090ce86 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:12:55.746907 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1d89681c624e0dad39685bd090ce86" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:12:55.747505 436 raft_consensus.cc:2454] T 2b1d89681c624e0dad39685bd090ce86 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:12:55.748656 32709 leader_election.cc:302] T 2b1d89681c624e0dad39685bd090ce86 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:12:55.749119 636 raft_consensus.cc:2781] T 2b1d89681c624e0dad39685bd090ce86 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:55.749272 636 raft_consensus.cc:482] T 2b1d89681c624e0dad39685bd090ce86 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:55.749362 636 raft_consensus.cc:3037] T 2b1d89681c624e0dad39685bd090ce86 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:55.751817 636 raft_consensus.cc:504] T 2b1d89681c624e0dad39685bd090ce86 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:55.752429 636 leader_election.cc:288] T 2b1d89681c624e0dad39685bd090ce86 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Requested vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:12:55.753258 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1d89681c624e0dad39685bd090ce86" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:12:55.753280 30006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1d89681c624e0dad39685bd090ce86" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:12:55.753469 436 raft_consensus.cc:3037] T 2b1d89681c624e0dad39685bd090ce86 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:55.753558 30006 raft_consensus.cc:2454] T 2b1d89681c624e0dad39685bd090ce86 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:12:55.753952 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1d89681c624e0dad39685bd090ce86" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:12:55.754196 30005 raft_consensus.cc:3037] T 2b1d89681c624e0dad39685bd090ce86 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:55.755772 436 raft_consensus.cc:2454] T 2b1d89681c624e0dad39685bd090ce86 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:12:55.757161 30005 raft_consensus.cc:2454] T 2b1d89681c624e0dad39685bd090ce86 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:12:55.757314 32709 leader_election.cc:302] T 2b1d89681c624e0dad39685bd090ce86 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:12:55.757632 636 raft_consensus.cc:2781] T 2b1d89681c624e0dad39685bd090ce86 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:55.765110 636 raft_consensus.cc:686] T 2b1d89681c624e0dad39685bd090ce86 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Becoming Leader. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Running, Role: LEADER
I20240503 02:12:55.765658 636 consensus_queue.cc:237] T 2b1d89681c624e0dad39685bd090ce86 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:55.770407 29733 catalog_manager.cc:5484] T 2b1d89681c624e0dad39685bd090ce86 P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6). New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:12:55.897464 29733 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_extra_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:55.897729 29727 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240503 02:12:55.898038 29727 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240503 02:12:55.903961 29727 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet c7f85d7037ce49a7b0bd9003d8977a5a
I20240503 02:12:55.905210 30114 tablet_service.cc:1514] Processing DeleteTablet for tablet c7f85d7037ce49a7b0bd9003d8977a5a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:55 UTC) from {username='slave'} at 127.0.0.1:44812
I20240503 02:12:55.905429 29731 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:12:55.906550 655 tablet_replica.cc:330] T c7f85d7037ce49a7b0bd9003d8977a5a P a9094240860e43bfb129193f3bfd5be6: stopping tablet replica
I20240503 02:12:55.906826 30246 tablet_service.cc:1514] Processing DeleteTablet for tablet c7f85d7037ce49a7b0bd9003d8977a5a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:55 UTC) from {username='slave'} at 127.0.0.1:49856
I20240503 02:12:55.906925 29984 tablet_service.cc:1514] Processing DeleteTablet for tablet c7f85d7037ce49a7b0bd9003d8977a5a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:55 UTC) from {username='slave'} at 127.0.0.1:52158
I20240503 02:12:55.907116 655 raft_consensus.cc:2229] T c7f85d7037ce49a7b0bd9003d8977a5a P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:55.907465 655 raft_consensus.cc:2258] T c7f85d7037ce49a7b0bd9003d8977a5a P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:55.908519 29727 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet 006b2e20b1b04b18bee60fcd822949c7
I20240503 02:12:55.909281 655 ts_tablet_manager.cc:1875] T c7f85d7037ce49a7b0bd9003d8977a5a P a9094240860e43bfb129193f3bfd5be6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:55.909459 29985 tablet_service.cc:1514] Processing DeleteTablet for tablet 006b2e20b1b04b18bee60fcd822949c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:55 UTC) from {username='slave'} at 127.0.0.1:52158
I20240503 02:12:55.913985 29853 tablet_service.cc:1514] Processing DeleteTablet for tablet 006b2e20b1b04b18bee60fcd822949c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:55 UTC) from {username='slave'} at 127.0.0.1:58744
I20240503 02:12:55.914350 30114 tablet_service.cc:1514] Processing DeleteTablet for tablet 006b2e20b1b04b18bee60fcd822949c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:55 UTC) from {username='slave'} at 127.0.0.1:44812
I20240503 02:12:55.914746 29727 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet c5bb83b2a415451aa6909503a47662dd
I20240503 02:12:55.915632 30244 tablet_service.cc:1514] Processing DeleteTablet for tablet c5bb83b2a415451aa6909503a47662dd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:55 UTC) from {username='slave'} at 127.0.0.1:49856
I20240503 02:12:55.915931 29985 tablet_service.cc:1514] Processing DeleteTablet for tablet c5bb83b2a415451aa6909503a47662dd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:55 UTC) from {username='slave'} at 127.0.0.1:52158
I20240503 02:12:55.916997 29854 tablet_service.cc:1514] Processing DeleteTablet for tablet c5bb83b2a415451aa6909503a47662dd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:55 UTC) from {username='slave'} at 127.0.0.1:58744
I20240503 02:12:55.917090 655 ts_tablet_manager.cc:1888] T c7f85d7037ce49a7b0bd9003d8977a5a P a9094240860e43bfb129193f3bfd5be6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:55.917385 655 log.cc:1196] T c7f85d7037ce49a7b0bd9003d8977a5a P a9094240860e43bfb129193f3bfd5be6: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/wals/c7f85d7037ce49a7b0bd9003d8977a5a
I20240503 02:12:55.918052 655 ts_tablet_manager.cc:1909] T c7f85d7037ce49a7b0bd9003d8977a5a P a9094240860e43bfb129193f3bfd5be6: Deleting consensus metadata
I20240503 02:12:55.919960 655 tablet_replica.cc:330] T 006b2e20b1b04b18bee60fcd822949c7 P a9094240860e43bfb129193f3bfd5be6: stopping tablet replica
I20240503 02:12:55.920058 29718 catalog_manager.cc:4830] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): tablet c7f85d7037ce49a7b0bd9003d8977a5a (table rebalance_test_table_0 [id=c53bfcf133b44207890453c637ba13d8]) successfully deleted
I20240503 02:12:55.920228 655 raft_consensus.cc:2229] T 006b2e20b1b04b18bee60fcd822949c7 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:55.920475 655 raft_consensus.cc:2258] T 006b2e20b1b04b18bee60fcd822949c7 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:55.921411 655 ts_tablet_manager.cc:1875] T 006b2e20b1b04b18bee60fcd822949c7 P a9094240860e43bfb129193f3bfd5be6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:55.926247 659 tablet_replica.cc:330] T 006b2e20b1b04b18bee60fcd822949c7 P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:12:55.926947 659 raft_consensus.cc:2229] T 006b2e20b1b04b18bee60fcd822949c7 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:55.927644 659 raft_consensus.cc:2258] T 006b2e20b1b04b18bee60fcd822949c7 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:55.928335 655 ts_tablet_manager.cc:1888] T 006b2e20b1b04b18bee60fcd822949c7 P a9094240860e43bfb129193f3bfd5be6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:55.928601 655 log.cc:1196] T 006b2e20b1b04b18bee60fcd822949c7 P a9094240860e43bfb129193f3bfd5be6: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/wals/006b2e20b1b04b18bee60fcd822949c7
I20240503 02:12:55.929174 655 ts_tablet_manager.cc:1909] T 006b2e20b1b04b18bee60fcd822949c7 P a9094240860e43bfb129193f3bfd5be6: Deleting consensus metadata
I20240503 02:12:55.929441 659 ts_tablet_manager.cc:1875] T 006b2e20b1b04b18bee60fcd822949c7 P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:55.930205 29718 catalog_manager.cc:4830] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): tablet 006b2e20b1b04b18bee60fcd822949c7 (table rebalance_test_table_0 [id=c53bfcf133b44207890453c637ba13d8]) successfully deleted
I20240503 02:12:55.936926 659 ts_tablet_manager.cc:1888] T 006b2e20b1b04b18bee60fcd822949c7 P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:55.937239 659 log.cc:1196] T 006b2e20b1b04b18bee60fcd822949c7 P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/006b2e20b1b04b18bee60fcd822949c7
I20240503 02:12:55.937588 657 tablet_replica.cc:330] T c7f85d7037ce49a7b0bd9003d8977a5a P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:12:55.937950 659 ts_tablet_manager.cc:1909] T 006b2e20b1b04b18bee60fcd822949c7 P 16440e365430405fb10366ab70e2b429: Deleting consensus metadata
I20240503 02:12:55.938139 30006 raft_consensus.cc:1261] T 2b1d89681c624e0dad39685bd090ce86 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:55.938238 657 raft_consensus.cc:2229] T c7f85d7037ce49a7b0bd9003d8977a5a P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:55.938577 657 raft_consensus.cc:2258] T c7f85d7037ce49a7b0bd9003d8977a5a P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:55.939682 436 raft_consensus.cc:1261] T 2b1d89681c624e0dad39685bd090ce86 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:55.939846 659 tablet_replica.cc:330] T c5bb83b2a415451aa6909503a47662dd P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:12:55.940162 659 raft_consensus.cc:2229] T c5bb83b2a415451aa6909503a47662dd P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:55.940296 657 ts_tablet_manager.cc:1875] T c7f85d7037ce49a7b0bd9003d8977a5a P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:55.940356 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet 006b2e20b1b04b18bee60fcd822949c7 (table rebalance_test_table_0 [id=c53bfcf133b44207890453c637ba13d8]) successfully deleted
I20240503 02:12:55.940503 659 raft_consensus.cc:2258] T c5bb83b2a415451aa6909503a47662dd P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:55.941498 659 ts_tablet_manager.cc:1875] T c5bb83b2a415451aa6909503a47662dd P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:55.942138 643 consensus_queue.cc:1035] T 2b1d89681c624e0dad39685bd090ce86 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:55.945676 643 consensus_queue.cc:1035] T 2b1d89681c624e0dad39685bd090ce86 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:55.949692 656 tablet_replica.cc:330] T c7f85d7037ce49a7b0bd9003d8977a5a P 1f10f80f13c84e80b03eda7085a3b012: stopping tablet replica
I20240503 02:12:55.950330 656 raft_consensus.cc:2229] T c7f85d7037ce49a7b0bd9003d8977a5a P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:55.951098 656 raft_consensus.cc:2258] T c7f85d7037ce49a7b0bd9003d8977a5a P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:55.953444 656 ts_tablet_manager.cc:1875] T c7f85d7037ce49a7b0bd9003d8977a5a P 1f10f80f13c84e80b03eda7085a3b012: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:55.959848 656 ts_tablet_manager.cc:1888] T c7f85d7037ce49a7b0bd9003d8977a5a P 1f10f80f13c84e80b03eda7085a3b012: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:55.960142 659 ts_tablet_manager.cc:1888] T c5bb83b2a415451aa6909503a47662dd P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:55.960378 659 log.cc:1196] T c5bb83b2a415451aa6909503a47662dd P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/c5bb83b2a415451aa6909503a47662dd
I20240503 02:12:55.961093 659 ts_tablet_manager.cc:1909] T c5bb83b2a415451aa6909503a47662dd P 16440e365430405fb10366ab70e2b429: Deleting consensus metadata
I20240503 02:12:55.962110 656 log.cc:1196] T c7f85d7037ce49a7b0bd9003d8977a5a P 1f10f80f13c84e80b03eda7085a3b012: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/wals/c7f85d7037ce49a7b0bd9003d8977a5a
I20240503 02:12:55.962153 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet c5bb83b2a415451aa6909503a47662dd (table rebalance_test_table_0 [id=c53bfcf133b44207890453c637ba13d8]) successfully deleted
I20240503 02:12:55.962756 656 ts_tablet_manager.cc:1909] T c7f85d7037ce49a7b0bd9003d8977a5a P 1f10f80f13c84e80b03eda7085a3b012: Deleting consensus metadata
I20240503 02:12:55.965011 656 tablet_replica.cc:330] T c5bb83b2a415451aa6909503a47662dd P 1f10f80f13c84e80b03eda7085a3b012: stopping tablet replica
I20240503 02:12:55.965322 656 raft_consensus.cc:2229] T c5bb83b2a415451aa6909503a47662dd P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:55.965714 656 raft_consensus.cc:2258] T c5bb83b2a415451aa6909503a47662dd P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:55.966476 656 ts_tablet_manager.cc:1875] T c5bb83b2a415451aa6909503a47662dd P 1f10f80f13c84e80b03eda7085a3b012: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:55.966945 657 ts_tablet_manager.cc:1888] T c7f85d7037ce49a7b0bd9003d8977a5a P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:55.968132 29719 catalog_manager.cc:4830] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): tablet c7f85d7037ce49a7b0bd9003d8977a5a (table rebalance_test_table_0 [id=c53bfcf133b44207890453c637ba13d8]) successfully deleted
I20240503 02:12:55.969275 417 tablet_service.cc:1467] Processing CreateTablet for tablet 737a39a4f65b4e9ab2ce65c0d7f62dcb (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=c9cc90e5637140278312ab93695a8afd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:55.969542 32755 tablet_service.cc:1467] Processing CreateTablet for tablet 5cf58a111e3048449303ec45ac091b42 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=c9cc90e5637140278312ab93695a8afd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:55.969895 417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 737a39a4f65b4e9ab2ce65c0d7f62dcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:55.970163 32755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cf58a111e3048449303ec45ac091b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:55.970808 657 log.cc:1196] T c7f85d7037ce49a7b0bd9003d8977a5a P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/c7f85d7037ce49a7b0bd9003d8977a5a
I20240503 02:12:55.971640 657 ts_tablet_manager.cc:1909] T c7f85d7037ce49a7b0bd9003d8977a5a P 155735c7f8b14997ac4c770aee2a9401: Deleting consensus metadata
I20240503 02:12:55.974035 29853 tablet_service.cc:1467] Processing CreateTablet for tablet 5cf58a111e3048449303ec45ac091b42 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=c9cc90e5637140278312ab93695a8afd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:55.974499 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cf58a111e3048449303ec45ac091b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:55.976342 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet c7f85d7037ce49a7b0bd9003d8977a5a (table rebalance_test_table_0 [id=c53bfcf133b44207890453c637ba13d8]) successfully deleted
I20240503 02:12:55.977042 657 tablet_replica.cc:330] T 006b2e20b1b04b18bee60fcd822949c7 P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:12:55.977622 657 raft_consensus.cc:2229] T 006b2e20b1b04b18bee60fcd822949c7 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:55.977612 416 tablet_service.cc:1467] Processing CreateTablet for tablet 83c7d1060ccf49a2adda4302600a9876 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=c9cc90e5637140278312ab93695a8afd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:55.978140 416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83c7d1060ccf49a2adda4302600a9876. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:55.979547 657 raft_consensus.cc:2258] T 006b2e20b1b04b18bee60fcd822949c7 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:55.981931 633 tablet_bootstrap.cc:492] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:12:55.984005 633 tablet_bootstrap.cc:654] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:55.985018 657 ts_tablet_manager.cc:1875] T 006b2e20b1b04b18bee60fcd822949c7 P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:55.989502 660 mvcc.cc:205] Tried to move back new op lower bound from 7023420931829145600 to 7023420931141111808. Current Snapshot: MvccSnapshot[applied={T|T < 7023420931829145600}]
I20240503 02:12:55.991590 29733 catalog_manager.cc:5576] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873) does not have the latest schema for tablet 2b1d89681c624e0dad39685bd090ce86 (table rebalancer_dynamic_table [id=19ebc3eaa8f54e818d1981b70d2b9f58]). Expected version 1 got 0
I20240503 02:12:56.000900 657 ts_tablet_manager.cc:1888] T 006b2e20b1b04b18bee60fcd822949c7 P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:56.005828 632 tablet_bootstrap.cc:492] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:12:56.006841 668 tablet.cc:1722] T 2b1d89681c624e0dad39685bd090ce86 P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:56.007566 632 tablet_bootstrap.cc:654] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:56.008150 668 tablet.cc:1620] T 2b1d89681c624e0dad39685bd090ce86 P 8a666e798fe8477d8dd1e51b385d5ead: MemRowSet was empty: no flush needed.
I20240503 02:12:56.015084 633 tablet_bootstrap.cc:492] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:12:56.015254 633 ts_tablet_manager.cc:1367] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.002s
I20240503 02:12:56.016338 633 raft_consensus.cc:348] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:56.016569 633 raft_consensus.cc:374] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:56.016670 633 raft_consensus.cc:729] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:12:56.016921 633 consensus_queue.cc:260] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:56.017668 633 ts_tablet_manager.cc:1398] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:12:56.021757 670 tablet_bootstrap.cc:492] T 5cf58a111e3048449303ec45ac091b42 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:56.022274 29727 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=19ebc3eaa8f54e818d1981b70d2b9f58] alter complete (version 1)
I20240503 02:12:56.024675 632 tablet_bootstrap.cc:492] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:12:56.025092 632 ts_tablet_manager.cc:1367] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.005s
I20240503 02:12:56.025574 656 ts_tablet_manager.cc:1888] T c5bb83b2a415451aa6909503a47662dd P 1f10f80f13c84e80b03eda7085a3b012: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:56.026582 632 raft_consensus.cc:348] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:56.026906 632 raft_consensus.cc:374] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:56.027025 632 raft_consensus.cc:729] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:12:56.027459 656 log.cc:1196] T c5bb83b2a415451aa6909503a47662dd P 1f10f80f13c84e80b03eda7085a3b012: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/wals/c5bb83b2a415451aa6909503a47662dd
I20240503 02:12:56.027395 632 consensus_queue.cc:260] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:56.028307 670 tablet_bootstrap.cc:654] T 5cf58a111e3048449303ec45ac091b42 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:56.033524 547 tablet_service.cc:1467] Processing CreateTablet for tablet 5cf58a111e3048449303ec45ac091b42 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=c9cc90e5637140278312ab93695a8afd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:56.034514 656 ts_tablet_manager.cc:1909] T c5bb83b2a415451aa6909503a47662dd P 1f10f80f13c84e80b03eda7085a3b012: Deleting consensus metadata
I20240503 02:12:56.034991 632 ts_tablet_manager.cc:1398] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 02:12:56.035145 547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cf58a111e3048449303ec45ac091b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:56.035403 632 tablet_bootstrap.cc:492] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:12:56.036897 29719 catalog_manager.cc:4830] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): tablet c5bb83b2a415451aa6909503a47662dd (table rebalance_test_table_0 [id=c53bfcf133b44207890453c637ba13d8]) successfully deleted
I20240503 02:12:56.037775 548 tablet_service.cc:1467] Processing CreateTablet for tablet 737a39a4f65b4e9ab2ce65c0d7f62dcb (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=c9cc90e5637140278312ab93695a8afd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:56.038286 548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 737a39a4f65b4e9ab2ce65c0d7f62dcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:56.039793 657 log.cc:1196] T 006b2e20b1b04b18bee60fcd822949c7 P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/006b2e20b1b04b18bee60fcd822949c7
I20240503 02:12:56.040308 657 ts_tablet_manager.cc:1909] T 006b2e20b1b04b18bee60fcd822949c7 P 155735c7f8b14997ac4c770aee2a9401: Deleting consensus metadata
I20240503 02:12:56.040314 670 tablet_bootstrap.cc:492] T 5cf58a111e3048449303ec45ac091b42 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:56.040472 670 ts_tablet_manager.cc:1367] T 5cf58a111e3048449303ec45ac091b42 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20240503 02:12:56.041245 632 tablet_bootstrap.cc:654] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:56.041303 670 raft_consensus.cc:348] T 5cf58a111e3048449303ec45ac091b42 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:56.041527 670 raft_consensus.cc:374] T 5cf58a111e3048449303ec45ac091b42 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:56.041630 670 raft_consensus.cc:729] T 5cf58a111e3048449303ec45ac091b42 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:56.041915 670 consensus_queue.cc:260] T 5cf58a111e3048449303ec45ac091b42 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:56.042469 670 ts_tablet_manager.cc:1398] T 5cf58a111e3048449303ec45ac091b42 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:56.034116 669 tablet.cc:1722] T 2b1d89681c624e0dad39685bd090ce86 P 155735c7f8b14997ac4c770aee2a9401: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:56.042892 32621 tablet_service.cc:1467] Processing CreateTablet for tablet 737a39a4f65b4e9ab2ce65c0d7f62dcb (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=c9cc90e5637140278312ab93695a8afd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:56.044358 32621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 737a39a4f65b4e9ab2ce65c0d7f62dcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:56.044417 669 tablet.cc:1620] T 2b1d89681c624e0dad39685bd090ce86 P 155735c7f8b14997ac4c770aee2a9401: MemRowSet was empty: no flush needed.
I20240503 02:12:56.045548 32620 tablet_service.cc:1467] Processing CreateTablet for tablet 83c7d1060ccf49a2adda4302600a9876 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=c9cc90e5637140278312ab93695a8afd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:56.045979 32620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83c7d1060ccf49a2adda4302600a9876. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:56.047334 29733 catalog_manager.cc:5576] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241) does not have the latest schema for tablet 2b1d89681c624e0dad39685bd090ce86 (table rebalancer_dynamic_table [id=19ebc3eaa8f54e818d1981b70d2b9f58]). Expected version 1 got 0
I20240503 02:12:56.047546 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet 006b2e20b1b04b18bee60fcd822949c7 (table rebalance_test_table_0 [id=c53bfcf133b44207890453c637ba13d8]) successfully deleted
I20240503 02:12:56.048710 657 tablet_replica.cc:330] T c5bb83b2a415451aa6909503a47662dd P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:12:56.048928 657 raft_consensus.cc:2229] T c5bb83b2a415451aa6909503a47662dd P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:56.049208 657 raft_consensus.cc:2258] T c5bb83b2a415451aa6909503a47662dd P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:56.049955 657 ts_tablet_manager.cc:1875] T c5bb83b2a415451aa6909503a47662dd P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:56.053323 546 tablet_service.cc:1467] Processing CreateTablet for tablet 83c7d1060ccf49a2adda4302600a9876 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=c9cc90e5637140278312ab93695a8afd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:56.054039 546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83c7d1060ccf49a2adda4302600a9876. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:56.059648 632 tablet_bootstrap.cc:492] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:12:56.059839 632 ts_tablet_manager.cc:1367] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.002s
I20240503 02:12:56.063179 29733 catalog_manager.cc:5576] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873) does not have the latest schema for tablet 2b1d89681c624e0dad39685bd090ce86 (table rebalancer_dynamic_table [id=19ebc3eaa8f54e818d1981b70d2b9f58]). Expected version 1 got 0
I20240503 02:12:56.062098 632 raft_consensus.cc:348] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:56.079995 632 raft_consensus.cc:374] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:56.080168 632 raft_consensus.cc:729] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:12:56.061589 657 ts_tablet_manager.cc:1888] T c5bb83b2a415451aa6909503a47662dd P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:56.080511 657 log.cc:1196] T c5bb83b2a415451aa6909503a47662dd P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/c5bb83b2a415451aa6909503a47662dd
I20240503 02:12:56.080667 632 consensus_queue.cc:260] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:56.081470 657 ts_tablet_manager.cc:1909] T c5bb83b2a415451aa6909503a47662dd P 155735c7f8b14997ac4c770aee2a9401: Deleting consensus metadata
I20240503 02:12:56.083240 632 ts_tablet_manager.cc:1398] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.023s user 0.002s sys 0.001s
I20240503 02:12:56.086043 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet c5bb83b2a415451aa6909503a47662dd (table rebalance_test_table_0 [id=c53bfcf133b44207890453c637ba13d8]) successfully deleted
I20240503 02:12:56.108777 678 tablet_bootstrap.cc:492] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:12:56.118711 678 tablet_bootstrap.cc:654] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:56.121623 675 tablet.cc:1722] T 2b1d89681c624e0dad39685bd090ce86 P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:56.123194 675 tablet.cc:1620] T 2b1d89681c624e0dad39685bd090ce86 P fbbc2be6382e47908ecb7c68c367bd26: MemRowSet was empty: no flush needed.
I20240503 02:12:56.123301 29727 catalog_manager.cc:5576] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241) does not have the latest schema for tablet 2b1d89681c624e0dad39685bd090ce86 (table rebalancer_dynamic_table [id=19ebc3eaa8f54e818d1981b70d2b9f58]). Expected version 1 got 0
I20240503 02:12:56.128154 678 log.cc:824] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P 9a4d92bc25374fb3960fde7d10be309d: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:56.128361 679 tablet_bootstrap.cc:492] T 83c7d1060ccf49a2adda4302600a9876 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:12:56.130990 678 tablet_bootstrap.cc:492] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:12:56.131273 678 ts_tablet_manager.cc:1367] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.000s
I20240503 02:12:56.132848 679 tablet_bootstrap.cc:654] T 83c7d1060ccf49a2adda4302600a9876 P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:56.135176 29733 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:12:56.136276 679 log.cc:824] T 83c7d1060ccf49a2adda4302600a9876 P f575ce92c0224eeb80b9fae5f6561fbe: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:56.145161 637 raft_consensus.cc:482] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:56.145401 637 raft_consensus.cc:504] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:56.145781 668 tablet.cc:1722] T 2b1d89681c624e0dad39685bd090ce86 P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
W20240503 02:12:56.146080 29921 tablet.cc:2369] T 5cf58a111e3048449303ec45ac091b42 P 16440e365430405fb10366ab70e2b429: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:12:56.146317 668 tablet.cc:1620] T 2b1d89681c624e0dad39685bd090ce86 P 8a666e798fe8477d8dd1e51b385d5ead: MemRowSet was empty: no flush needed.
I20240503 02:12:56.147346 637 leader_election.cc:288] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:12:56.148694 669 tablet.cc:1722] T 2b1d89681c624e0dad39685bd090ce86 P 155735c7f8b14997ac4c770aee2a9401: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:12:56.149210 669 tablet.cc:1620] T 2b1d89681c624e0dad39685bd090ce86 P 155735c7f8b14997ac4c770aee2a9401: MemRowSet was empty: no flush needed.
I20240503 02:12:56.155627 675 tablet.cc:1722] T 2b1d89681c624e0dad39685bd090ce86 P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:12:56.156565 675 tablet.cc:1620] T 2b1d89681c624e0dad39685bd090ce86 P fbbc2be6382e47908ecb7c68c367bd26: MemRowSet was empty: no flush needed.
I20240503 02:12:56.157567 29733 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=19ebc3eaa8f54e818d1981b70d2b9f58] alter complete (version 2)
I20240503 02:12:56.165197 678 raft_consensus.cc:348] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:56.166970 678 raft_consensus.cc:374] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:56.167101 678 raft_consensus.cc:729] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:12:56.168084 678 consensus_queue.cc:260] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:56.172487 643 raft_consensus.cc:482] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:56.172688 643 raft_consensus.cc:504] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:56.173713 643 leader_election.cc:288] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:56.173949 639 raft_consensus.cc:482] T 5cf58a111e3048449303ec45ac091b42 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:56.174124 639 raft_consensus.cc:504] T 5cf58a111e3048449303ec45ac091b42 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:56.175046 639 leader_election.cc:288] T 5cf58a111e3048449303ec45ac091b42 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:12:56.175566 679 tablet_bootstrap.cc:492] T 83c7d1060ccf49a2adda4302600a9876 P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:12:56.176061 679 ts_tablet_manager.cc:1367] T 83c7d1060ccf49a2adda4302600a9876 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.048s user 0.007s sys 0.009s
I20240503 02:12:56.177211 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737a39a4f65b4e9ab2ce65c0d7f62dcb" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:12:56.177758 32641 raft_consensus.cc:2454] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 0.
I20240503 02:12:56.178848 371 leader_election.cc:302] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:12:56.179343 637 raft_consensus.cc:2781] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:56.179450 637 raft_consensus.cc:482] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:56.179536 637 raft_consensus.cc:3037] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:56.182667 637 raft_consensus.cc:504] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:56.183866 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737a39a4f65b4e9ab2ce65c0d7f62dcb" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d"
I20240503 02:12:56.184074 32641 raft_consensus.cc:3037] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:56.185937 679 raft_consensus.cc:348] T 83c7d1060ccf49a2adda4302600a9876 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:56.186287 568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737a39a4f65b4e9ab2ce65c0d7f62dcb" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:12:56.186468 679 raft_consensus.cc:374] T 83c7d1060ccf49a2adda4302600a9876 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:56.186584 679 raft_consensus.cc:729] T 83c7d1060ccf49a2adda4302600a9876 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:12:56.187376 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737a39a4f65b4e9ab2ce65c0d7f62dcb" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:12:56.187711 32641 raft_consensus.cc:2454] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 1.
W20240503 02:12:56.188175 370 leader_election.cc:341] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:56.188742 370 leader_election.cc:341] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:56.189134 371 leader_election.cc:302] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, fbbc2be6382e47908ecb7c68c367bd26; no voters: f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:12:56.183359 637 leader_election.cc:288] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Requested vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:12:56.189746 637 raft_consensus.cc:2781] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:56.189957 637 raft_consensus.cc:686] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Becoming Leader. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Running, Role: LEADER
I20240503 02:12:56.189968 679 consensus_queue.cc:260] T 83c7d1060ccf49a2adda4302600a9876 P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:56.190546 637 consensus_queue.cc:237] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:56.199002 679 ts_tablet_manager.cc:1398] T 83c7d1060ccf49a2adda4302600a9876 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.023s user 0.014s sys 0.004s
I20240503 02:12:56.199018 692 raft_consensus.cc:482] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:56.199360 692 raft_consensus.cc:504] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:56.199589 613 heartbeater.cc:503] Master 127.22.108.62:40071 was elected leader, sending a full tablet report...
I20240503 02:12:56.200381 692 leader_election.cc:288] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:12:56.202565 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c7d1060ccf49a2adda4302600a9876" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:12:56.202970 567 raft_consensus.cc:2454] T 83c7d1060ccf49a2adda4302600a9876 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 0.
I20240503 02:12:56.203435 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c7d1060ccf49a2adda4302600a9876" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:12:56.204993 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cf58a111e3048449303ec45ac091b42" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
W20240503 02:12:56.205564 371 leader_election.cc:341] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:56.205693 32708 leader_election.cc:341] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:56.205701 29733 catalog_manager.cc:5484] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7). New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:12:56.205886 370 leader_election.cc:302] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f575ce92c0224eeb80b9fae5f6561fbe, fbbc2be6382e47908ecb7c68c367bd26; no voters: 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:12:56.206287 637 raft_consensus.cc:2781] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:56.206413 679 tablet_bootstrap.cc:492] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:12:56.206410 637 raft_consensus.cc:482] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:56.206499 637 raft_consensus.cc:3037] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:56.208199 679 tablet_bootstrap.cc:654] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:56.212591 637 raft_consensus.cc:504] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:56.213367 637 leader_election.cc:288] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Requested vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:12:56.213515 679 tablet_bootstrap.cc:492] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:12:56.213733 679 ts_tablet_manager.cc:1367] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20240503 02:12:56.213884 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c7d1060ccf49a2adda4302600a9876" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:12:56.214123 567 raft_consensus.cc:3037] T 83c7d1060ccf49a2adda4302600a9876 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:56.214478 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c7d1060ccf49a2adda4302600a9876" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d"
W20240503 02:12:56.215068 371 leader_election.cc:341] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:56.215061 679 raft_consensus.cc:348] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:56.215330 679 raft_consensus.cc:374] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:56.215444 679 raft_consensus.cc:729] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:12:56.215742 679 consensus_queue.cc:260] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:56.216426 679 ts_tablet_manager.cc:1398] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 02:12:56.216747 679 tablet_bootstrap.cc:492] T 5cf58a111e3048449303ec45ac091b42 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:12:56.218382 567 raft_consensus.cc:2454] T 83c7d1060ccf49a2adda4302600a9876 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 1.
I20240503 02:12:56.218683 679 tablet_bootstrap.cc:654] T 5cf58a111e3048449303ec45ac091b42 P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:56.219252 370 leader_election.cc:302] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f575ce92c0224eeb80b9fae5f6561fbe, fbbc2be6382e47908ecb7c68c367bd26; no voters: 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:12:56.219743 637 raft_consensus.cc:2781] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:56.219895 637 raft_consensus.cc:686] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Becoming Leader. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Running, Role: LEADER
I20240503 02:12:56.220194 637 consensus_queue.cc:237] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:56.228296 29733 catalog_manager.cc:5484] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7). New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:12:56.249298 678 ts_tablet_manager.cc:1398] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.118s user 0.013s sys 0.004s
I20240503 02:12:56.249853 678 tablet_bootstrap.cc:492] T 83c7d1060ccf49a2adda4302600a9876 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:12:56.251991 678 tablet_bootstrap.cc:654] T 83c7d1060ccf49a2adda4302600a9876 P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:56.256506 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cf58a111e3048449303ec45ac091b42" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:12:56.256783 29874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cf58a111e3048449303ec45ac091b42" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:56.257062 29874 raft_consensus.cc:2454] T 5cf58a111e3048449303ec45ac091b42 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:12:56.257705 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cf58a111e3048449303ec45ac091b42" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:12:56.258121 679 tablet_bootstrap.cc:492] T 5cf58a111e3048449303ec45ac091b42 P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:12:56.258121 678 tablet_bootstrap.cc:492] T 83c7d1060ccf49a2adda4302600a9876 P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:12:56.258265 678 ts_tablet_manager.cc:1367] T 83c7d1060ccf49a2adda4302600a9876 P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.002s
I20240503 02:12:56.258283 679 ts_tablet_manager.cc:1367] T 5cf58a111e3048449303ec45ac091b42 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.042s user 0.010s sys 0.016s
I20240503 02:12:56.258213 307 raft_consensus.cc:2454] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
W20240503 02:12:56.258921 29808 leader_election.cc:341] T 5cf58a111e3048449303ec45ac091b42 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:56.259025 679 raft_consensus.cc:348] T 5cf58a111e3048449303ec45ac091b42 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:56.259259 679 raft_consensus.cc:374] T 5cf58a111e3048449303ec45ac091b42 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:56.259338 679 raft_consensus.cc:729] T 5cf58a111e3048449303ec45ac091b42 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:12:56.259232 678 raft_consensus.cc:348] T 83c7d1060ccf49a2adda4302600a9876 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:56.259440 678 raft_consensus.cc:374] T 83c7d1060ccf49a2adda4302600a9876 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:56.259593 679 consensus_queue.cc:260] T 5cf58a111e3048449303ec45ac091b42 P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:56.260020 32711 leader_election.cc:302] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16440e365430405fb10366ab70e2b429, 8a666e798fe8477d8dd1e51b385d5ead; no voters: f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:12:56.260275 679 ts_tablet_manager.cc:1398] T 5cf58a111e3048449303ec45ac091b42 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:56.260501 643 raft_consensus.cc:2781] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:56.260617 643 raft_consensus.cc:482] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:56.260699 643 raft_consensus.cc:3037] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:56.262007 29727 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:12:56.263473 643 raft_consensus.cc:504] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:56.265933 29809 leader_election.cc:302] T 5cf58a111e3048449303ec45ac091b42 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16440e365430405fb10366ab70e2b429, 8a666e798fe8477d8dd1e51b385d5ead; no voters: f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:12:56.266526 639 raft_consensus.cc:2781] T 5cf58a111e3048449303ec45ac091b42 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:56.266654 639 raft_consensus.cc:482] T 5cf58a111e3048449303ec45ac091b42 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:56.266741 639 raft_consensus.cc:3037] T 5cf58a111e3048449303ec45ac091b42 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:56.266827 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cf58a111e3048449303ec45ac091b42" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:12:56.267076 567 raft_consensus.cc:3037] T 5cf58a111e3048449303ec45ac091b42 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:56.269505 639 raft_consensus.cc:504] T 5cf58a111e3048449303ec45ac091b42 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:56.270164 639 leader_election.cc:288] T 5cf58a111e3048449303ec45ac091b42 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Requested vote from peers 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:12:56.270947 568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cf58a111e3048449303ec45ac091b42" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:12:56.271257 678 raft_consensus.cc:729] T 83c7d1060ccf49a2adda4302600a9876 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:12:56.272071 678 consensus_queue.cc:260] T 83c7d1060ccf49a2adda4302600a9876 P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:56.272686 678 ts_tablet_manager.cc:1398] T 83c7d1060ccf49a2adda4302600a9876 P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20240503 02:12:56.273687 29874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cf58a111e3048449303ec45ac091b42" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:12:56.273927 29874 raft_consensus.cc:2379] T 5cf58a111e3048449303ec45ac091b42 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a666e798fe8477d8dd1e51b385d5ead in current term 1: Already voted for candidate 16440e365430405fb10366ab70e2b429 in this term.
I20240503 02:12:56.273941 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cf58a111e3048449303ec45ac091b42" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:12:56.274230 307 raft_consensus.cc:2379] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 16440e365430405fb10366ab70e2b429 in current term 1: Already voted for candidate 8a666e798fe8477d8dd1e51b385d5ead in this term.
I20240503 02:12:56.274961 29809 leader_election.cc:302] T 5cf58a111e3048449303ec45ac091b42 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 16440e365430405fb10366ab70e2b429; no voters: 8a666e798fe8477d8dd1e51b385d5ead, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:12:56.275667 639 raft_consensus.cc:2726] T 5cf58a111e3048449303ec45ac091b42 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 02:12:56.277002 567 raft_consensus.cc:2454] T 5cf58a111e3048449303ec45ac091b42 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:12:56.277743 668 tablet.cc:1722] T 2b1d89681c624e0dad39685bd090ce86 P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:12:56.279153 32708 leader_election.cc:302] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, f575ce92c0224eeb80b9fae5f6561fbe; no voters: 16440e365430405fb10366ab70e2b429
I20240503 02:12:56.284476 675 tablet.cc:1722] T 2b1d89681c624e0dad39685bd090ce86 P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:12:56.285218 708 raft_consensus.cc:2781] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:56.286669 643 leader_election.cc:288] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Requested vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:56.288244 708 raft_consensus.cc:686] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Becoming Leader. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Running, Role: LEADER
I20240503 02:12:56.288765 708 consensus_queue.cc:237] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:56.301079 29727 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table_new_name [id=19ebc3eaa8f54e818d1981b70d2b9f58] alter complete (version 3)
I20240503 02:12:56.302316 669 tablet.cc:1722] T 2b1d89681c624e0dad39685bd090ce86 P 155735c7f8b14997ac4c770aee2a9401: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:12:56.303478 29727 catalog_manager.cc:5484] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6). New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:12:56.318682 693 raft_consensus.cc:482] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:56.318877 693 raft_consensus.cc:504] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:56.320619 693 leader_election.cc:288] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:12:56.345701 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737a39a4f65b4e9ab2ce65c0d7f62dcb" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:12:56.346089 32641 raft_consensus.cc:2379] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f575ce92c0224eeb80b9fae5f6561fbe in current term 1: Already voted for candidate fbbc2be6382e47908ecb7c68c367bd26 in this term.
I20240503 02:12:56.354179 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737a39a4f65b4e9ab2ce65c0d7f62dcb" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
W20240503 02:12:56.354543 32688 tablet.cc:2369] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P 9a4d92bc25374fb3960fde7d10be309d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:12:56.355161 502 leader_election.cc:302] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f575ce92c0224eeb80b9fae5f6561fbe; no voters: 9a4d92bc25374fb3960fde7d10be309d, fbbc2be6382e47908ecb7c68c367bd26
I20240503 02:12:56.355612 693 raft_consensus.cc:3037] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:56.358613 693 raft_consensus.cc:2726] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 02:12:56.398818 614 tablet.cc:2369] T 5cf58a111e3048449303ec45ac091b42 P f575ce92c0224eeb80b9fae5f6561fbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:12:56.409143 29729 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:12:56.409487 29729 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:12:56.413437 29729 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet 2b1d89681c624e0dad39685bd090ce86
I20240503 02:12:56.414678 417 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b1d89681c624e0dad39685bd090ce86 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:56 UTC) from {username='slave'} at 127.0.0.1:50814
I20240503 02:12:56.414783 29984 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b1d89681c624e0dad39685bd090ce86 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:56 UTC) from {username='slave'} at 127.0.0.1:52158
I20240503 02:12:56.415612 657 tablet_replica.cc:330] T 2b1d89681c624e0dad39685bd090ce86 P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:12:56.415855 657 raft_consensus.cc:2229] T 2b1d89681c624e0dad39685bd090ce86 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:56.416065 657 raft_consensus.cc:2258] T 2b1d89681c624e0dad39685bd090ce86 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:56.416494 32755 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b1d89681c624e0dad39685bd090ce86 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:56 UTC) from {username='slave'} at 127.0.0.1:46360
I20240503 02:12:56.417547 738 tablet_replica.cc:330] T 2b1d89681c624e0dad39685bd090ce86 P 8a666e798fe8477d8dd1e51b385d5ead: stopping tablet replica
I20240503 02:12:56.418035 738 raft_consensus.cc:2229] T 2b1d89681c624e0dad39685bd090ce86 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:56.418689 738 raft_consensus.cc:2258] T 2b1d89681c624e0dad39685bd090ce86 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:56.418852 737 tablet_replica.cc:330] T 2b1d89681c624e0dad39685bd090ce86 P fbbc2be6382e47908ecb7c68c367bd26: stopping tablet replica
I20240503 02:12:56.419602 657 ts_tablet_manager.cc:1875] T 2b1d89681c624e0dad39685bd090ce86 P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:56.419845 737 raft_consensus.cc:2229] T 2b1d89681c624e0dad39685bd090ce86 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:56.420125 738 ts_tablet_manager.cc:1875] T 2b1d89681c624e0dad39685bd090ce86 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:56.420279 737 raft_consensus.cc:2258] T 2b1d89681c624e0dad39685bd090ce86 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:56.421890 737 ts_tablet_manager.cc:1875] T 2b1d89681c624e0dad39685bd090ce86 P fbbc2be6382e47908ecb7c68c367bd26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:56.425374 29728 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240503 02:12:56.425591 29728 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240503 02:12:56.429993 737 ts_tablet_manager.cc:1888] T 2b1d89681c624e0dad39685bd090ce86 P fbbc2be6382e47908ecb7c68c367bd26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:56.430151 737 log.cc:1196] T 2b1d89681c624e0dad39685bd090ce86 P fbbc2be6382e47908ecb7c68c367bd26: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal/wals/2b1d89681c624e0dad39685bd090ce86
I20240503 02:12:56.430773 737 ts_tablet_manager.cc:1909] T 2b1d89681c624e0dad39685bd090ce86 P fbbc2be6382e47908ecb7c68c367bd26: Deleting consensus metadata
I20240503 02:12:56.434738 29728 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet 70820e67bc3740a08bcf0510a055c544
I20240503 02:12:56.435117 29728 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet 8706c9b491b54500963bb81e6e459ddd
I20240503 02:12:56.435611 29728 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet bd91cf40fd654426a11789f37c4dca47
I20240503 02:12:56.435660 29853 tablet_service.cc:1514] Processing DeleteTablet for tablet 70820e67bc3740a08bcf0510a055c544 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:56 UTC) from {username='slave'} at 127.0.0.1:58744
I20240503 02:12:56.435791 30114 tablet_service.cc:1514] Processing DeleteTablet for tablet 8706c9b491b54500963bb81e6e459ddd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:56 UTC) from {username='slave'} at 127.0.0.1:44812
I20240503 02:12:56.435958 659 tablet_replica.cc:330] T 70820e67bc3740a08bcf0510a055c544 P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:12:56.436167 659 raft_consensus.cc:2229] T 70820e67bc3740a08bcf0510a055c544 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:56.436472 659 raft_consensus.cc:2258] T 70820e67bc3740a08bcf0510a055c544 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:56.437106 739 tablet_replica.cc:330] T 8706c9b491b54500963bb81e6e459ddd P a9094240860e43bfb129193f3bfd5be6: stopping tablet replica
I20240503 02:12:56.437232 659 ts_tablet_manager.cc:1875] T 70820e67bc3740a08bcf0510a055c544 P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:56.437391 739 raft_consensus.cc:2229] T 8706c9b491b54500963bb81e6e459ddd P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:56.439983 29985 tablet_service.cc:1514] Processing DeleteTablet for tablet 8706c9b491b54500963bb81e6e459ddd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:56 UTC) from {username='slave'} at 127.0.0.1:52158
I20240503 02:12:56.440124 29984 tablet_service.cc:1514] Processing DeleteTablet for tablet 70820e67bc3740a08bcf0510a055c544 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:56 UTC) from {username='slave'} at 127.0.0.1:52158
I20240503 02:12:56.440321 29986 tablet_service.cc:1514] Processing DeleteTablet for tablet bd91cf40fd654426a11789f37c4dca47 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:56 UTC) from {username='slave'} at 127.0.0.1:52158
I20240503 02:12:56.439429 739 raft_consensus.cc:2258] T 8706c9b491b54500963bb81e6e459ddd P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:56.440701 30246 tablet_service.cc:1514] Processing DeleteTablet for tablet 70820e67bc3740a08bcf0510a055c544 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:56 UTC) from {username='slave'} at 127.0.0.1:49856
I20240503 02:12:56.441046 30245 tablet_service.cc:1514] Processing DeleteTablet for tablet bd91cf40fd654426a11789f37c4dca47 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:56 UTC) from {username='slave'} at 127.0.0.1:49856
I20240503 02:12:56.441170 656 tablet_replica.cc:330] T 70820e67bc3740a08bcf0510a055c544 P 1f10f80f13c84e80b03eda7085a3b012: stopping tablet replica
I20240503 02:12:56.441362 656 raft_consensus.cc:2229] T 70820e67bc3740a08bcf0510a055c544 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:56.441555 739 ts_tablet_manager.cc:1875] T 8706c9b491b54500963bb81e6e459ddd P a9094240860e43bfb129193f3bfd5be6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:56.441766 656 raft_consensus.cc:2258] T 70820e67bc3740a08bcf0510a055c544 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:56.441890 738 ts_tablet_manager.cc:1888] T 2b1d89681c624e0dad39685bd090ce86 P 8a666e798fe8477d8dd1e51b385d5ead: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:56.441974 29719 catalog_manager.cc:4830] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): tablet 2b1d89681c624e0dad39685bd090ce86 (table rebalancer_dynamic_table_new_name [id=19ebc3eaa8f54e818d1981b70d2b9f58]) successfully deleted
I20240503 02:12:56.442039 738 log.cc:1196] T 2b1d89681c624e0dad39685bd090ce86 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal/wals/2b1d89681c624e0dad39685bd090ce86
I20240503 02:12:56.442550 656 ts_tablet_manager.cc:1875] T 70820e67bc3740a08bcf0510a055c544 P 1f10f80f13c84e80b03eda7085a3b012: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:56.442589 738 ts_tablet_manager.cc:1909] T 2b1d89681c624e0dad39685bd090ce86 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting consensus metadata
I20240503 02:12:56.444331 29719 catalog_manager.cc:4830] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): tablet 2b1d89681c624e0dad39685bd090ce86 (table rebalancer_dynamic_table_new_name [id=19ebc3eaa8f54e818d1981b70d2b9f58]) successfully deleted
I20240503 02:12:56.440985 30244 tablet_service.cc:1514] Processing DeleteTablet for tablet 8706c9b491b54500963bb81e6e459ddd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:56 UTC) from {username='slave'} at 127.0.0.1:49856
I20240503 02:12:56.446337 657 ts_tablet_manager.cc:1888] T 2b1d89681c624e0dad39685bd090ce86 P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:56.446630 657 log.cc:1196] T 2b1d89681c624e0dad39685bd090ce86 P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/2b1d89681c624e0dad39685bd090ce86
I20240503 02:12:56.447355 29853 tablet_service.cc:1514] Processing DeleteTablet for tablet bd91cf40fd654426a11789f37c4dca47 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:56 UTC) from {username='slave'} at 127.0.0.1:58744
I20240503 02:12:56.448622 657 ts_tablet_manager.cc:1909] T 2b1d89681c624e0dad39685bd090ce86 P 155735c7f8b14997ac4c770aee2a9401: Deleting consensus metadata
I20240503 02:12:56.451145 739 ts_tablet_manager.cc:1888] T 8706c9b491b54500963bb81e6e459ddd P a9094240860e43bfb129193f3bfd5be6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:56.451329 739 log.cc:1196] T 8706c9b491b54500963bb81e6e459ddd P a9094240860e43bfb129193f3bfd5be6: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/wals/8706c9b491b54500963bb81e6e459ddd
I20240503 02:12:56.451918 739 ts_tablet_manager.cc:1909] T 8706c9b491b54500963bb81e6e459ddd P a9094240860e43bfb129193f3bfd5be6: Deleting consensus metadata
I20240503 02:12:56.453011 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet 2b1d89681c624e0dad39685bd090ce86 (table rebalancer_dynamic_table_new_name [id=19ebc3eaa8f54e818d1981b70d2b9f58]) successfully deleted
I20240503 02:12:56.453403 657 tablet_replica.cc:330] T 8706c9b491b54500963bb81e6e459ddd P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:12:56.453626 657 raft_consensus.cc:2229] T 8706c9b491b54500963bb81e6e459ddd P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:56.453866 657 raft_consensus.cc:2258] T 8706c9b491b54500963bb81e6e459ddd P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:56.454558 657 ts_tablet_manager.cc:1875] T 8706c9b491b54500963bb81e6e459ddd P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:56.454620 29718 catalog_manager.cc:4830] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): tablet 8706c9b491b54500963bb81e6e459ddd (table rebalance_test_table_2 [id=2de6ddd95bd0493cac0dd361e1642e8f]) successfully deleted
I20240503 02:12:56.461534 657 ts_tablet_manager.cc:1888] T 8706c9b491b54500963bb81e6e459ddd P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:56.461721 657 log.cc:1196] T 8706c9b491b54500963bb81e6e459ddd P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/8706c9b491b54500963bb81e6e459ddd
I20240503 02:12:56.462203 657 ts_tablet_manager.cc:1909] T 8706c9b491b54500963bb81e6e459ddd P 155735c7f8b14997ac4c770aee2a9401: Deleting consensus metadata
I20240503 02:12:56.475068 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet 8706c9b491b54500963bb81e6e459ddd (table rebalance_test_table_2 [id=2de6ddd95bd0493cac0dd361e1642e8f]) successfully deleted
I20240503 02:12:56.475488 657 tablet_replica.cc:330] T 70820e67bc3740a08bcf0510a055c544 P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:12:56.475766 657 raft_consensus.cc:2229] T 70820e67bc3740a08bcf0510a055c544 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:56.476023 657 raft_consensus.cc:2258] T 70820e67bc3740a08bcf0510a055c544 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:56.478688 656 ts_tablet_manager.cc:1888] T 70820e67bc3740a08bcf0510a055c544 P 1f10f80f13c84e80b03eda7085a3b012: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:56.480450 657 ts_tablet_manager.cc:1875] T 70820e67bc3740a08bcf0510a055c544 P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:56.478829 656 log.cc:1196] T 70820e67bc3740a08bcf0510a055c544 P 1f10f80f13c84e80b03eda7085a3b012: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/wals/70820e67bc3740a08bcf0510a055c544
I20240503 02:12:56.480976 656 ts_tablet_manager.cc:1909] T 70820e67bc3740a08bcf0510a055c544 P 1f10f80f13c84e80b03eda7085a3b012: Deleting consensus metadata
I20240503 02:12:56.480533 659 ts_tablet_manager.cc:1888] T 70820e67bc3740a08bcf0510a055c544 P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:56.482007 659 log.cc:1196] T 70820e67bc3740a08bcf0510a055c544 P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/70820e67bc3740a08bcf0510a055c544
I20240503 02:12:56.483332 659 ts_tablet_manager.cc:1909] T 70820e67bc3740a08bcf0510a055c544 P 16440e365430405fb10366ab70e2b429: Deleting consensus metadata
I20240503 02:12:56.484171 29719 catalog_manager.cc:4830] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): tablet 70820e67bc3740a08bcf0510a055c544 (table rebalance_test_table_2 [id=2de6ddd95bd0493cac0dd361e1642e8f]) successfully deleted
I20240503 02:12:56.484944 656 tablet_replica.cc:330] T bd91cf40fd654426a11789f37c4dca47 P 1f10f80f13c84e80b03eda7085a3b012: stopping tablet replica
I20240503 02:12:56.485306 656 raft_consensus.cc:2229] T bd91cf40fd654426a11789f37c4dca47 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:56.486868 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet 70820e67bc3740a08bcf0510a055c544 (table rebalance_test_table_2 [id=2de6ddd95bd0493cac0dd361e1642e8f]) successfully deleted
I20240503 02:12:56.487212 656 raft_consensus.cc:2258] T bd91cf40fd654426a11789f37c4dca47 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:56.487493 659 tablet_replica.cc:330] T bd91cf40fd654426a11789f37c4dca47 P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:12:56.487704 659 raft_consensus.cc:2229] T bd91cf40fd654426a11789f37c4dca47 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:56.487922 659 raft_consensus.cc:2258] T bd91cf40fd654426a11789f37c4dca47 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:56.489107 656 ts_tablet_manager.cc:1875] T bd91cf40fd654426a11789f37c4dca47 P 1f10f80f13c84e80b03eda7085a3b012: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:56.491916 657 ts_tablet_manager.cc:1888] T 70820e67bc3740a08bcf0510a055c544 P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:56.492067 657 log.cc:1196] T 70820e67bc3740a08bcf0510a055c544 P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/70820e67bc3740a08bcf0510a055c544
I20240503 02:12:56.492516 657 ts_tablet_manager.cc:1909] T 70820e67bc3740a08bcf0510a055c544 P 155735c7f8b14997ac4c770aee2a9401: Deleting consensus metadata
I20240503 02:12:56.493613 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet 70820e67bc3740a08bcf0510a055c544 (table rebalance_test_table_2 [id=2de6ddd95bd0493cac0dd361e1642e8f]) successfully deleted
I20240503 02:12:56.493978 657 tablet_replica.cc:330] T bd91cf40fd654426a11789f37c4dca47 P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:12:56.494177 657 raft_consensus.cc:2229] T bd91cf40fd654426a11789f37c4dca47 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:56.495033 657 raft_consensus.cc:2258] T bd91cf40fd654426a11789f37c4dca47 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:56.495802 657 ts_tablet_manager.cc:1875] T bd91cf40fd654426a11789f37c4dca47 P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:56.500270 659 ts_tablet_manager.cc:1875] T bd91cf40fd654426a11789f37c4dca47 P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:56.500946 657 ts_tablet_manager.cc:1888] T bd91cf40fd654426a11789f37c4dca47 P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:56.501119 657 log.cc:1196] T bd91cf40fd654426a11789f37c4dca47 P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/bd91cf40fd654426a11789f37c4dca47
I20240503 02:12:56.501485 657 ts_tablet_manager.cc:1909] T bd91cf40fd654426a11789f37c4dca47 P 155735c7f8b14997ac4c770aee2a9401: Deleting consensus metadata
I20240503 02:12:56.502305 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet bd91cf40fd654426a11789f37c4dca47 (table rebalance_test_table_2 [id=2de6ddd95bd0493cac0dd361e1642e8f]) successfully deleted
I20240503 02:12:56.506625 656 ts_tablet_manager.cc:1888] T bd91cf40fd654426a11789f37c4dca47 P 1f10f80f13c84e80b03eda7085a3b012: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:56.506770 656 log.cc:1196] T bd91cf40fd654426a11789f37c4dca47 P 1f10f80f13c84e80b03eda7085a3b012: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/wals/bd91cf40fd654426a11789f37c4dca47
I20240503 02:12:56.507144 656 ts_tablet_manager.cc:1909] T bd91cf40fd654426a11789f37c4dca47 P 1f10f80f13c84e80b03eda7085a3b012: Deleting consensus metadata
I20240503 02:12:56.507949 656 tablet_replica.cc:330] T 8706c9b491b54500963bb81e6e459ddd P 1f10f80f13c84e80b03eda7085a3b012: stopping tablet replica
I20240503 02:12:56.508015 687 raft_consensus.cc:482] T 83c7d1060ccf49a2adda4302600a9876 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:56.508097 29719 catalog_manager.cc:4830] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): tablet bd91cf40fd654426a11789f37c4dca47 (table rebalance_test_table_2 [id=2de6ddd95bd0493cac0dd361e1642e8f]) successfully deleted
I20240503 02:12:56.508159 656 raft_consensus.cc:2229] T 8706c9b491b54500963bb81e6e459ddd P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:56.508402 656 raft_consensus.cc:2258] T 8706c9b491b54500963bb81e6e459ddd P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:56.508239 687 raft_consensus.cc:504] T 83c7d1060ccf49a2adda4302600a9876 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:56.508981 656 ts_tablet_manager.cc:1875] T 8706c9b491b54500963bb81e6e459ddd P 1f10f80f13c84e80b03eda7085a3b012: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:56.510833 687 leader_election.cc:288] T 83c7d1060ccf49a2adda4302600a9876 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:12:56.515870 656 ts_tablet_manager.cc:1888] T 8706c9b491b54500963bb81e6e459ddd P 1f10f80f13c84e80b03eda7085a3b012: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:56.516033 656 log.cc:1196] T 8706c9b491b54500963bb81e6e459ddd P 1f10f80f13c84e80b03eda7085a3b012: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/wals/8706c9b491b54500963bb81e6e459ddd
I20240503 02:12:56.516530 656 ts_tablet_manager.cc:1909] T 8706c9b491b54500963bb81e6e459ddd P 1f10f80f13c84e80b03eda7085a3b012: Deleting consensus metadata
I20240503 02:12:56.516824 29733 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:12:56.518913 29719 catalog_manager.cc:4830] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): tablet 8706c9b491b54500963bb81e6e459ddd (table rebalance_test_table_2 [id=2de6ddd95bd0493cac0dd361e1642e8f]) successfully deleted
I20240503 02:12:56.526823 659 ts_tablet_manager.cc:1888] T bd91cf40fd654426a11789f37c4dca47 P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:56.526995 659 log.cc:1196] T bd91cf40fd654426a11789f37c4dca47 P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/bd91cf40fd654426a11789f37c4dca47
I20240503 02:12:56.527478 659 ts_tablet_manager.cc:1909] T bd91cf40fd654426a11789f37c4dca47 P 16440e365430405fb10366ab70e2b429: Deleting consensus metadata
I20240503 02:12:56.528267 30244 tablet_service.cc:1467] Processing CreateTablet for tablet 7c1b368d5f8d46f1a5e56a3ff746e77b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=17fce000d51e45689be7e25cb26c8067]), partition=
I20240503 02:12:56.528414 30114 tablet_service.cc:1467] Processing CreateTablet for tablet 7c1b368d5f8d46f1a5e56a3ff746e77b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=17fce000d51e45689be7e25cb26c8067]), partition=
I20240503 02:12:56.528681 30244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c1b368d5f8d46f1a5e56a3ff746e77b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:56.528915 30114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c1b368d5f8d46f1a5e56a3ff746e77b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:56.533458 417 tablet_service.cc:1467] Processing CreateTablet for tablet 7c1b368d5f8d46f1a5e56a3ff746e77b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=17fce000d51e45689be7e25cb26c8067]), partition=
I20240503 02:12:56.533900 417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c1b368d5f8d46f1a5e56a3ff746e77b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:56.545610 632 tablet_bootstrap.cc:492] T 7c1b368d5f8d46f1a5e56a3ff746e77b P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:12:56.547796 632 tablet_bootstrap.cc:654] T 7c1b368d5f8d46f1a5e56a3ff746e77b P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:56.551751 747 tablet_bootstrap.cc:492] T 7c1b368d5f8d46f1a5e56a3ff746e77b P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:56.553915 747 tablet_bootstrap.cc:654] T 7c1b368d5f8d46f1a5e56a3ff746e77b P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:56.555593 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet bd91cf40fd654426a11789f37c4dca47 (table rebalance_test_table_2 [id=2de6ddd95bd0493cac0dd361e1642e8f]) successfully deleted
I20240503 02:12:56.559267 632 tablet_bootstrap.cc:492] T 7c1b368d5f8d46f1a5e56a3ff746e77b P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:12:56.559461 632 ts_tablet_manager.cc:1367] T 7c1b368d5f8d46f1a5e56a3ff746e77b P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.006s
I20240503 02:12:56.560488 632 raft_consensus.cc:348] T 7c1b368d5f8d46f1a5e56a3ff746e77b P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:56.560788 632 raft_consensus.cc:374] T 7c1b368d5f8d46f1a5e56a3ff746e77b P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:56.560897 632 raft_consensus.cc:729] T 7c1b368d5f8d46f1a5e56a3ff746e77b P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:12:56.565253 632 consensus_queue.cc:260] T 7c1b368d5f8d46f1a5e56a3ff746e77b P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:56.565941 632 ts_tablet_manager.cc:1398] T 7c1b368d5f8d46f1a5e56a3ff746e77b P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20240503 02:12:56.566591 745 tablet_bootstrap.cc:492] T 7c1b368d5f8d46f1a5e56a3ff746e77b P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:12:56.568642 745 tablet_bootstrap.cc:654] T 7c1b368d5f8d46f1a5e56a3ff746e77b P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:56.576936 745 tablet_bootstrap.cc:492] T 7c1b368d5f8d46f1a5e56a3ff746e77b P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:12:56.577570 745 ts_tablet_manager.cc:1367] T 7c1b368d5f8d46f1a5e56a3ff746e77b P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.004s
I20240503 02:12:56.579617 745 raft_consensus.cc:348] T 7c1b368d5f8d46f1a5e56a3ff746e77b P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:56.580607 745 raft_consensus.cc:374] T 7c1b368d5f8d46f1a5e56a3ff746e77b P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:56.580893 745 raft_consensus.cc:729] T 7c1b368d5f8d46f1a5e56a3ff746e77b P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:12:56.581413 745 consensus_queue.cc:260] T 7c1b368d5f8d46f1a5e56a3ff746e77b P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:56.587168 637 consensus_queue.cc:1035] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:56.596657 32487 raft_consensus.cc:482] T 7c1b368d5f8d46f1a5e56a3ff746e77b P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:56.596845 32487 raft_consensus.cc:504] T 7c1b368d5f8d46f1a5e56a3ff746e77b P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:56.597656 32487 leader_election.cc:288] T 7c1b368d5f8d46f1a5e56a3ff746e77b P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:12:56.599344 568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c7d1060ccf49a2adda4302600a9876" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:12:56.599550 745 ts_tablet_manager.cc:1398] T 7c1b368d5f8d46f1a5e56a3ff746e77b P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.022s user 0.000s sys 0.002s
I20240503 02:12:56.599625 568 raft_consensus.cc:2379] T 83c7d1060ccf49a2adda4302600a9876 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a4d92bc25374fb3960fde7d10be309d in current term 1: Already voted for candidate fbbc2be6382e47908ecb7c68c367bd26 in this term.
I20240503 02:12:56.609464 29853 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:12:56.629421 637 consensus_queue.cc:1035] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:56.635615 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c7d1060ccf49a2adda4302600a9876" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:12:56.636574 32574 leader_election.cc:302] T 83c7d1060ccf49a2adda4302600a9876 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d; no voters: f575ce92c0224eeb80b9fae5f6561fbe, fbbc2be6382e47908ecb7c68c367bd26
I20240503 02:12:56.637707 687 raft_consensus.cc:3037] T 83c7d1060ccf49a2adda4302600a9876 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:56.642215 687 raft_consensus.cc:2726] T 83c7d1060ccf49a2adda4302600a9876 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:56.649075 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c1b368d5f8d46f1a5e56a3ff746e77b" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
W20240503 02:12:56.651858 30070 leader_election.cc:341] T 7c1b368d5f8d46f1a5e56a3ff746e77b P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:56.656605 692 consensus_queue.cc:1035] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:56.657637 747 tablet_bootstrap.cc:492] T 7c1b368d5f8d46f1a5e56a3ff746e77b P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:56.658914 747 ts_tablet_manager.cc:1367] T 7c1b368d5f8d46f1a5e56a3ff746e77b P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.107s user 0.001s sys 0.003s
I20240503 02:12:56.659936 747 raft_consensus.cc:348] T 7c1b368d5f8d46f1a5e56a3ff746e77b P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:56.660269 747 raft_consensus.cc:374] T 7c1b368d5f8d46f1a5e56a3ff746e77b P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:56.660419 747 raft_consensus.cc:729] T 7c1b368d5f8d46f1a5e56a3ff746e77b P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:56.660802 747 consensus_queue.cc:260] T 7c1b368d5f8d46f1a5e56a3ff746e77b P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:56.663522 747 ts_tablet_manager.cc:1398] T 7c1b368d5f8d46f1a5e56a3ff746e77b P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:12:56.668288 32755 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:56.669196 29986 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:12:56.670583 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c1b368d5f8d46f1a5e56a3ff746e77b" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:12:56.670840 437 raft_consensus.cc:2454] T 7c1b368d5f8d46f1a5e56a3ff746e77b P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 0.
I20240503 02:12:56.671083 708 consensus_queue.cc:1035] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:56.671223 30070 leader_election.cc:302] T 7c1b368d5f8d46f1a5e56a3ff746e77b P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6, fbbc2be6382e47908ecb7c68c367bd26; no voters: 1f10f80f13c84e80b03eda7085a3b012
I20240503 02:12:56.671633 32487 raft_consensus.cc:2781] T 7c1b368d5f8d46f1a5e56a3ff746e77b P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:56.671802 32487 raft_consensus.cc:482] T 7c1b368d5f8d46f1a5e56a3ff746e77b P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:56.671900 32487 raft_consensus.cc:3037] T 7c1b368d5f8d46f1a5e56a3ff746e77b P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:56.673887 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:12:56.674723 32487 raft_consensus.cc:504] T 7c1b368d5f8d46f1a5e56a3ff746e77b P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:56.677836 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c1b368d5f8d46f1a5e56a3ff746e77b" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:12:56.678064 30264 raft_consensus.cc:3037] T 7c1b368d5f8d46f1a5e56a3ff746e77b P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:56.678232 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c1b368d5f8d46f1a5e56a3ff746e77b" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:12:56.678436 437 raft_consensus.cc:3037] T 7c1b368d5f8d46f1a5e56a3ff746e77b P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:56.680986 30264 raft_consensus.cc:2454] T 7c1b368d5f8d46f1a5e56a3ff746e77b P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 1.
I20240503 02:12:56.681259 437 raft_consensus.cc:2454] T 7c1b368d5f8d46f1a5e56a3ff746e77b P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 1.
I20240503 02:12:56.681746 30070 leader_election.cc:302] T 7c1b368d5f8d46f1a5e56a3ff746e77b P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:12:56.682126 32703 raft_consensus.cc:2781] T 7c1b368d5f8d46f1a5e56a3ff746e77b P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:56.682154 32487 leader_election.cc:288] T 7c1b368d5f8d46f1a5e56a3ff746e77b P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Requested vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:12:56.688331 709 consensus_queue.cc:1035] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:56.689978 30114 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:12:56.700901 32703 raft_consensus.cc:686] T 7c1b368d5f8d46f1a5e56a3ff746e77b P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Becoming Leader. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Running, Role: LEADER
I20240503 02:12:56.701462 32703 consensus_queue.cc:237] T 7c1b368d5f8d46f1a5e56a3ff746e77b P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:12:56.710795 29733 catalog_manager.cc:5484] T 7c1b368d5f8d46f1a5e56a3ff746e77b P a9094240860e43bfb129193f3bfd5be6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9094240860e43bfb129193f3bfd5be6 (127.22.108.3). New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:12:56.721071 692 consensus_queue.cc:1035] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:56.741441 32620 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:56.742347 546 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:56.764974 417 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 02:12:56.769515 30312 tablet.cc:2369] T 7c1b368d5f8d46f1a5e56a3ff746e77b P 1f10f80f13c84e80b03eda7085a3b012: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:12:56.815136 30182 tablet.cc:2369] T 7c1b368d5f8d46f1a5e56a3ff746e77b P a9094240860e43bfb129193f3bfd5be6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:12:56.851989 29733 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:12:56.854795 29730 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_extra_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:56.867066 30244 tablet_service.cc:1467] Processing CreateTablet for tablet 00aedc4902a745d6a859cbbf99529a7f (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=27940b10e50e4a418d9e7155f5a25394]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:56.867614 30244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00aedc4902a745d6a859cbbf99529a7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:56.870318 32755 tablet_service.cc:1467] Processing CreateTablet for tablet 00aedc4902a745d6a859cbbf99529a7f (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=27940b10e50e4a418d9e7155f5a25394]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:56.870633 32621 tablet_service.cc:1467] Processing CreateTablet for tablet 5ccde7688c4e49b5b9d89d6e048c308e (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=27940b10e50e4a418d9e7155f5a25394]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:56.870746 32755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00aedc4902a745d6a859cbbf99529a7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:56.871031 32621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ccde7688c4e49b5b9d89d6e048c308e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:56.871870 417 tablet_service.cc:1467] Processing CreateTablet for tablet 5ccde7688c4e49b5b9d89d6e048c308e (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=27940b10e50e4a418d9e7155f5a25394]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:56.872311 417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ccde7688c4e49b5b9d89d6e048c308e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:56.873370 32620 tablet_service.cc:1467] Processing CreateTablet for tablet 00aedc4902a745d6a859cbbf99529a7f (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=27940b10e50e4a418d9e7155f5a25394]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:56.873461 32753 tablet_service.cc:1467] Processing CreateTablet for tablet 9be5395489a3488da62f4d5b66facf3d (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=27940b10e50e4a418d9e7155f5a25394]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:56.873752 32620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00aedc4902a745d6a859cbbf99529a7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:56.873831 32753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9be5395489a3488da62f4d5b66facf3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:56.876471 32754 tablet_service.cc:1467] Processing CreateTablet for tablet 5ccde7688c4e49b5b9d89d6e048c308e (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=27940b10e50e4a418d9e7155f5a25394]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:56.876914 32754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ccde7688c4e49b5b9d89d6e048c308e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:56.877475 32619 tablet_service.cc:1467] Processing CreateTablet for tablet 9be5395489a3488da62f4d5b66facf3d (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=27940b10e50e4a418d9e7155f5a25394]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:56.877944 32619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9be5395489a3488da62f4d5b66facf3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:56.868883 546 tablet_service.cc:1467] Processing CreateTablet for tablet 9be5395489a3488da62f4d5b66facf3d (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=27940b10e50e4a418d9e7155f5a25394]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:56.878396 546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9be5395489a3488da62f4d5b66facf3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:56.899863 779 tablet_bootstrap.cc:492] T 5ccde7688c4e49b5b9d89d6e048c308e P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:12:56.908751 779 tablet_bootstrap.cc:654] T 5ccde7688c4e49b5b9d89d6e048c308e P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:56.910091 747 tablet_bootstrap.cc:492] T 00aedc4902a745d6a859cbbf99529a7f P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:56.910423 632 tablet_bootstrap.cc:492] T 5ccde7688c4e49b5b9d89d6e048c308e P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:12:56.911770 747 tablet_bootstrap.cc:654] T 00aedc4902a745d6a859cbbf99529a7f P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:56.912108 632 tablet_bootstrap.cc:654] T 5ccde7688c4e49b5b9d89d6e048c308e P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:56.918623 780 tablet_bootstrap.cc:492] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:12:56.921164 777 tablet_bootstrap.cc:492] T 5ccde7688c4e49b5b9d89d6e048c308e P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:12:56.923671 777 tablet_bootstrap.cc:654] T 5ccde7688c4e49b5b9d89d6e048c308e P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:56.924481 779 tablet_bootstrap.cc:492] T 5ccde7688c4e49b5b9d89d6e048c308e P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:12:56.924714 779 ts_tablet_manager.cc:1367] T 5ccde7688c4e49b5b9d89d6e048c308e P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.000s
I20240503 02:12:56.925745 779 raft_consensus.cc:348] T 5ccde7688c4e49b5b9d89d6e048c308e P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:56.926908 779 raft_consensus.cc:374] T 5ccde7688c4e49b5b9d89d6e048c308e P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:56.927062 779 raft_consensus.cc:729] T 5ccde7688c4e49b5b9d89d6e048c308e P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:12:56.927485 779 consensus_queue.cc:260] T 5ccde7688c4e49b5b9d89d6e048c308e P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:56.928187 780 tablet_bootstrap.cc:654] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:56.929296 779 ts_tablet_manager.cc:1398] T 5ccde7688c4e49b5b9d89d6e048c308e P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:56.931109 777 tablet_bootstrap.cc:492] T 5ccde7688c4e49b5b9d89d6e048c308e P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:12:56.932199 437 raft_consensus.cc:1261] T 7c1b368d5f8d46f1a5e56a3ff746e77b P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:56.932322 30264 raft_consensus.cc:1261] T 7c1b368d5f8d46f1a5e56a3ff746e77b P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:56.932811 32703 consensus_queue.cc:1035] T 7c1b368d5f8d46f1a5e56a3ff746e77b P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:56.934990 779 tablet_bootstrap.cc:492] T 00aedc4902a745d6a859cbbf99529a7f P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:12:56.935640 632 tablet_bootstrap.cc:492] T 5ccde7688c4e49b5b9d89d6e048c308e P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:12:56.935776 632 ts_tablet_manager.cc:1367] T 5ccde7688c4e49b5b9d89d6e048c308e P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.007s
I20240503 02:12:56.936069 32703 consensus_queue.cc:1035] T 7c1b368d5f8d46f1a5e56a3ff746e77b P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:56.936429 632 raft_consensus.cc:348] T 5ccde7688c4e49b5b9d89d6e048c308e P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:56.936684 632 raft_consensus.cc:374] T 5ccde7688c4e49b5b9d89d6e048c308e P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:56.936755 632 raft_consensus.cc:729] T 5ccde7688c4e49b5b9d89d6e048c308e P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:12:56.936906 777 ts_tablet_manager.cc:1367] T 5ccde7688c4e49b5b9d89d6e048c308e P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240503 02:12:56.938129 777 raft_consensus.cc:348] T 5ccde7688c4e49b5b9d89d6e048c308e P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:56.938972 29730 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240503 02:12:56.939205 29730 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240503 02:12:56.945395 777 raft_consensus.cc:374] T 5ccde7688c4e49b5b9d89d6e048c308e P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:56.946157 779 tablet_bootstrap.cc:654] T 00aedc4902a745d6a859cbbf99529a7f P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:56.946388 777 raft_consensus.cc:729] T 5ccde7688c4e49b5b9d89d6e048c308e P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:12:56.946866 29730 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet 723a18a040ec41fa9d06459f4db808c7
I20240503 02:12:56.947134 777 consensus_queue.cc:260] T 5ccde7688c4e49b5b9d89d6e048c308e P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:56.948472 29853 tablet_service.cc:1514] Processing DeleteTablet for tablet 723a18a040ec41fa9d06459f4db808c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:56 UTC) from {username='slave'} at 127.0.0.1:58744
I20240503 02:12:56.948917 659 tablet_replica.cc:330] T 723a18a040ec41fa9d06459f4db808c7 P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:12:56.949172 659 raft_consensus.cc:2229] T 723a18a040ec41fa9d06459f4db808c7 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:56.949441 659 raft_consensus.cc:2258] T 723a18a040ec41fa9d06459f4db808c7 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:56.950171 659 ts_tablet_manager.cc:1875] T 723a18a040ec41fa9d06459f4db808c7 P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:56.950177 777 ts_tablet_manager.cc:1398] T 5ccde7688c4e49b5b9d89d6e048c308e P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240503 02:12:56.950500 777 tablet_bootstrap.cc:492] T 9be5395489a3488da62f4d5b66facf3d P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:12:56.954427 779 tablet_bootstrap.cc:492] T 00aedc4902a745d6a859cbbf99529a7f P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:12:56.954623 779 ts_tablet_manager.cc:1367] T 00aedc4902a745d6a859cbbf99529a7f P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20240503 02:12:56.955689 29733 catalog_manager.cc:5576] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241) does not have the latest schema for tablet 7c1b368d5f8d46f1a5e56a3ff746e77b (table rebalancer_dynamic_table [id=17fce000d51e45689be7e25cb26c8067]). Expected version 1 got 0
I20240503 02:12:56.957010 777 tablet_bootstrap.cc:654] T 9be5395489a3488da62f4d5b66facf3d P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:56.959154 779 raft_consensus.cc:348] T 00aedc4902a745d6a859cbbf99529a7f P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:56.959393 779 raft_consensus.cc:374] T 00aedc4902a745d6a859cbbf99529a7f P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:56.959470 779 raft_consensus.cc:729] T 00aedc4902a745d6a859cbbf99529a7f P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:12:56.960081 661 mvcc.cc:205] Tried to move back new op lower bound from 7023420935901159424 to 7023420934972694528. Current Snapshot: MvccSnapshot[applied={T|T < 7023420935901159424}]
I20240503 02:12:56.960026 794 tablet.cc:1722] T 7c1b368d5f8d46f1a5e56a3ff746e77b P a9094240860e43bfb129193f3bfd5be6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:56.960621 29986 tablet_service.cc:1514] Processing DeleteTablet for tablet 723a18a040ec41fa9d06459f4db808c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:56 UTC) from {username='slave'} at 127.0.0.1:52158
I20240503 02:12:56.960705 30114 tablet_service.cc:1514] Processing DeleteTablet for tablet 723a18a040ec41fa9d06459f4db808c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:56 UTC) from {username='slave'} at 127.0.0.1:44812
I20240503 02:12:56.960888 657 tablet_replica.cc:330] T 723a18a040ec41fa9d06459f4db808c7 P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:12:56.961189 657 raft_consensus.cc:2229] T 723a18a040ec41fa9d06459f4db808c7 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:56.961419 796 tablet_replica.cc:330] T 723a18a040ec41fa9d06459f4db808c7 P a9094240860e43bfb129193f3bfd5be6: stopping tablet replica
I20240503 02:12:56.961469 29730 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet 537c249dd4a44c23ade69a960b6e2675
I20240503 02:12:56.961621 657 raft_consensus.cc:2258] T 723a18a040ec41fa9d06459f4db808c7 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:56.961627 796 raft_consensus.cc:2229] T 723a18a040ec41fa9d06459f4db808c7 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:56.961946 796 raft_consensus.cc:2258] T 723a18a040ec41fa9d06459f4db808c7 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:56.962097 29986 tablet_service.cc:1514] Processing DeleteTablet for tablet 537c249dd4a44c23ade69a960b6e2675 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:56 UTC) from {username='slave'} at 127.0.0.1:52158
I20240503 02:12:56.962431 657 ts_tablet_manager.cc:1875] T 723a18a040ec41fa9d06459f4db808c7 P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:56.962692 796 ts_tablet_manager.cc:1875] T 723a18a040ec41fa9d06459f4db808c7 P a9094240860e43bfb129193f3bfd5be6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:56.964704 30244 tablet_service.cc:1514] Processing DeleteTablet for tablet 537c249dd4a44c23ade69a960b6e2675 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:56 UTC) from {username='slave'} at 127.0.0.1:49856
I20240503 02:12:56.965108 656 tablet_replica.cc:330] T 537c249dd4a44c23ade69a960b6e2675 P 1f10f80f13c84e80b03eda7085a3b012: stopping tablet replica
I20240503 02:12:56.965345 656 raft_consensus.cc:2229] T 537c249dd4a44c23ade69a960b6e2675 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:56.965577 656 raft_consensus.cc:2258] T 537c249dd4a44c23ade69a960b6e2675 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:56.965885 29853 tablet_service.cc:1514] Processing DeleteTablet for tablet 537c249dd4a44c23ade69a960b6e2675 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:56 UTC) from {username='slave'} at 127.0.0.1:58744
I20240503 02:12:56.966295 656 ts_tablet_manager.cc:1875] T 537c249dd4a44c23ade69a960b6e2675 P 1f10f80f13c84e80b03eda7085a3b012: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:56.966250 29730 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet fa8e2df1f59e4b86a5c18067108e07bf
I20240503 02:12:56.967043 30244 tablet_service.cc:1514] Processing DeleteTablet for tablet fa8e2df1f59e4b86a5c18067108e07bf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:56 UTC) from {username='slave'} at 127.0.0.1:49856
I20240503 02:12:56.967969 29853 tablet_service.cc:1514] Processing DeleteTablet for tablet fa8e2df1f59e4b86a5c18067108e07bf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:56 UTC) from {username='slave'} at 127.0.0.1:58744
I20240503 02:12:56.968713 30114 tablet_service.cc:1514] Processing DeleteTablet for tablet fa8e2df1f59e4b86a5c18067108e07bf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:56 UTC) from {username='slave'} at 127.0.0.1:44812
I20240503 02:12:56.969988 657 ts_tablet_manager.cc:1888] T 723a18a040ec41fa9d06459f4db808c7 P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:56.970150 657 log.cc:1196] T 723a18a040ec41fa9d06459f4db808c7 P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/723a18a040ec41fa9d06459f4db808c7
I20240503 02:12:56.970611 657 ts_tablet_manager.cc:1909] T 723a18a040ec41fa9d06459f4db808c7 P 155735c7f8b14997ac4c770aee2a9401: Deleting consensus metadata
I20240503 02:12:56.971616 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet 723a18a040ec41fa9d06459f4db808c7 (table rebalance_test_table_4 [id=e0ea42a974fb456485b832e1a5cb54ef]) successfully deleted
I20240503 02:12:56.971706 657 tablet_replica.cc:330] T 537c249dd4a44c23ade69a960b6e2675 P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:12:56.971905 657 raft_consensus.cc:2229] T 537c249dd4a44c23ade69a960b6e2675 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:56.972126 657 raft_consensus.cc:2258] T 537c249dd4a44c23ade69a960b6e2675 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:56.972172 632 consensus_queue.cc:260] T 5ccde7688c4e49b5b9d89d6e048c308e P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:56.973192 657 ts_tablet_manager.cc:1875] T 537c249dd4a44c23ade69a960b6e2675 P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:56.974771 632 ts_tablet_manager.cc:1398] T 5ccde7688c4e49b5b9d89d6e048c308e P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.039s user 0.000s sys 0.003s
I20240503 02:12:56.975553 795 tablet.cc:1722] T 7c1b368d5f8d46f1a5e56a3ff746e77b P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:56.975656 659 ts_tablet_manager.cc:1888] T 723a18a040ec41fa9d06459f4db808c7 P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:56.975801 659 log.cc:1196] T 723a18a040ec41fa9d06459f4db808c7 P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/723a18a040ec41fa9d06459f4db808c7
I20240503 02:12:56.976155 795 tablet.cc:1620] T 7c1b368d5f8d46f1a5e56a3ff746e77b P fbbc2be6382e47908ecb7c68c367bd26: MemRowSet was empty: no flush needed.
I20240503 02:12:56.977365 794 tablet.cc:1620] T 7c1b368d5f8d46f1a5e56a3ff746e77b P a9094240860e43bfb129193f3bfd5be6: MemRowSet was empty: no flush needed.
I20240503 02:12:56.980497 659 ts_tablet_manager.cc:1909] T 723a18a040ec41fa9d06459f4db808c7 P 16440e365430405fb10366ab70e2b429: Deleting consensus metadata
I20240503 02:12:56.982100 29730 catalog_manager.cc:5576] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241) does not have the latest schema for tablet 7c1b368d5f8d46f1a5e56a3ff746e77b (table rebalancer_dynamic_table [id=17fce000d51e45689be7e25cb26c8067]). Expected version 1 got 0
I20240503 02:12:56.983006 656 ts_tablet_manager.cc:1888] T 537c249dd4a44c23ade69a960b6e2675 P 1f10f80f13c84e80b03eda7085a3b012: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:56.983184 656 log.cc:1196] T 537c249dd4a44c23ade69a960b6e2675 P 1f10f80f13c84e80b03eda7085a3b012: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/wals/537c249dd4a44c23ade69a960b6e2675
I20240503 02:12:56.983814 656 ts_tablet_manager.cc:1909] T 537c249dd4a44c23ade69a960b6e2675 P 1f10f80f13c84e80b03eda7085a3b012: Deleting consensus metadata
I20240503 02:12:56.984948 656 tablet_replica.cc:330] T fa8e2df1f59e4b86a5c18067108e07bf P 1f10f80f13c84e80b03eda7085a3b012: stopping tablet replica
I20240503 02:12:56.985276 656 raft_consensus.cc:2229] T fa8e2df1f59e4b86a5c18067108e07bf P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:56.985301 29719 catalog_manager.cc:4830] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): tablet 537c249dd4a44c23ade69a960b6e2675 (table rebalance_test_table_4 [id=e0ea42a974fb456485b832e1a5cb54ef]) successfully deleted
I20240503 02:12:56.985687 656 raft_consensus.cc:2258] T fa8e2df1f59e4b86a5c18067108e07bf P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:56.985913 796 ts_tablet_manager.cc:1888] T 723a18a040ec41fa9d06459f4db808c7 P a9094240860e43bfb129193f3bfd5be6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:56.986629 656 ts_tablet_manager.cc:1875] T fa8e2df1f59e4b86a5c18067108e07bf P 1f10f80f13c84e80b03eda7085a3b012: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:56.990041 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet 723a18a040ec41fa9d06459f4db808c7 (table rebalance_test_table_4 [id=e0ea42a974fb456485b832e1a5cb54ef]) successfully deleted
I20240503 02:12:56.991616 747 tablet_bootstrap.cc:492] T 00aedc4902a745d6a859cbbf99529a7f P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:56.991950 657 ts_tablet_manager.cc:1888] T 537c249dd4a44c23ade69a960b6e2675 P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:56.992079 657 log.cc:1196] T 537c249dd4a44c23ade69a960b6e2675 P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/537c249dd4a44c23ade69a960b6e2675
I20240503 02:12:56.992499 657 ts_tablet_manager.cc:1909] T 537c249dd4a44c23ade69a960b6e2675 P 155735c7f8b14997ac4c770aee2a9401: Deleting consensus metadata
I20240503 02:12:56.993072 747 ts_tablet_manager.cc:1367] T 00aedc4902a745d6a859cbbf99529a7f P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.083s user 0.019s sys 0.010s
I20240503 02:12:56.993135 659 tablet_replica.cc:330] T 537c249dd4a44c23ade69a960b6e2675 P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:12:56.993557 659 raft_consensus.cc:2229] T 537c249dd4a44c23ade69a960b6e2675 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:56.994082 747 raft_consensus.cc:348] T 00aedc4902a745d6a859cbbf99529a7f P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:56.994217 659 raft_consensus.cc:2258] T 537c249dd4a44c23ade69a960b6e2675 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:56.994391 747 raft_consensus.cc:374] T 00aedc4902a745d6a859cbbf99529a7f P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:56.994526 747 raft_consensus.cc:729] T 00aedc4902a745d6a859cbbf99529a7f P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:56.994961 747 consensus_queue.cc:260] T 00aedc4902a745d6a859cbbf99529a7f P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:56.995842 747 ts_tablet_manager.cc:1398] T 00aedc4902a745d6a859cbbf99529a7f P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:56.996210 796 log.cc:1196] T 723a18a040ec41fa9d06459f4db808c7 P a9094240860e43bfb129193f3bfd5be6: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/wals/723a18a040ec41fa9d06459f4db808c7
I20240503 02:12:56.997431 796 ts_tablet_manager.cc:1909] T 723a18a040ec41fa9d06459f4db808c7 P a9094240860e43bfb129193f3bfd5be6: Deleting consensus metadata
I20240503 02:12:56.997675 659 ts_tablet_manager.cc:1875] T 537c249dd4a44c23ade69a960b6e2675 P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:57.000821 29718 catalog_manager.cc:4830] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): tablet 723a18a040ec41fa9d06459f4db808c7 (table rebalance_test_table_4 [id=e0ea42a974fb456485b832e1a5cb54ef]) successfully deleted
I20240503 02:12:57.001255 796 tablet_replica.cc:330] T fa8e2df1f59e4b86a5c18067108e07bf P a9094240860e43bfb129193f3bfd5be6: stopping tablet replica
I20240503 02:12:57.001533 796 raft_consensus.cc:2229] T fa8e2df1f59e4b86a5c18067108e07bf P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:57.002130 29730 catalog_manager.cc:5576] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935) does not have the latest schema for tablet 7c1b368d5f8d46f1a5e56a3ff746e77b (table rebalancer_dynamic_table [id=17fce000d51e45689be7e25cb26c8067]). Expected version 1 got 0
I20240503 02:12:57.002487 796 raft_consensus.cc:2258] T fa8e2df1f59e4b86a5c18067108e07bf P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:57.002295 800 tablet.cc:1722] T 7c1b368d5f8d46f1a5e56a3ff746e77b P 1f10f80f13c84e80b03eda7085a3b012: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:57.004230 780 tablet_bootstrap.cc:492] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:12:57.004396 780 ts_tablet_manager.cc:1367] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.086s user 0.006s sys 0.000s
I20240503 02:12:57.005257 780 raft_consensus.cc:348] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:57.005473 780 raft_consensus.cc:374] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:57.005579 780 raft_consensus.cc:729] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:12:57.005872 780 consensus_queue.cc:260] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:57.006101 796 ts_tablet_manager.cc:1875] T fa8e2df1f59e4b86a5c18067108e07bf P a9094240860e43bfb129193f3bfd5be6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:57.006435 780 ts_tablet_manager.cc:1398] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:57.006922 29733 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=17fce000d51e45689be7e25cb26c8067] alter complete (version 1)
I20240503 02:12:57.009526 779 consensus_queue.cc:260] T 00aedc4902a745d6a859cbbf99529a7f P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:57.018505 796 ts_tablet_manager.cc:1888] T fa8e2df1f59e4b86a5c18067108e07bf P a9094240860e43bfb129193f3bfd5be6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:57.018779 796 log.cc:1196] T fa8e2df1f59e4b86a5c18067108e07bf P a9094240860e43bfb129193f3bfd5be6: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/wals/fa8e2df1f59e4b86a5c18067108e07bf
I20240503 02:12:57.019208 777 tablet_bootstrap.cc:492] T 9be5395489a3488da62f4d5b66facf3d P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:12:57.019333 796 ts_tablet_manager.cc:1909] T fa8e2df1f59e4b86a5c18067108e07bf P a9094240860e43bfb129193f3bfd5be6: Deleting consensus metadata
I20240503 02:12:57.019398 777 ts_tablet_manager.cc:1367] T 9be5395489a3488da62f4d5b66facf3d P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.069s user 0.001s sys 0.003s
I20240503 02:12:57.020097 659 ts_tablet_manager.cc:1888] T 537c249dd4a44c23ade69a960b6e2675 P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:57.020321 659 log.cc:1196] T 537c249dd4a44c23ade69a960b6e2675 P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/537c249dd4a44c23ade69a960b6e2675
I20240503 02:12:57.020202 777 raft_consensus.cc:348] T 9be5395489a3488da62f4d5b66facf3d P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:57.020426 777 raft_consensus.cc:374] T 9be5395489a3488da62f4d5b66facf3d P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:57.020463 800 tablet.cc:1620] T 7c1b368d5f8d46f1a5e56a3ff746e77b P 1f10f80f13c84e80b03eda7085a3b012: MemRowSet was empty: no flush needed.
I20240503 02:12:57.020546 777 raft_consensus.cc:729] T 9be5395489a3488da62f4d5b66facf3d P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:12:57.020545 29718 catalog_manager.cc:4830] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): tablet fa8e2df1f59e4b86a5c18067108e07bf (table rebalance_test_table_4 [id=e0ea42a974fb456485b832e1a5cb54ef]) successfully deleted
I20240503 02:12:57.018816 779 ts_tablet_manager.cc:1398] T 00aedc4902a745d6a859cbbf99529a7f P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.064s user 0.003s sys 0.000s
I20240503 02:12:57.021101 659 ts_tablet_manager.cc:1909] T 537c249dd4a44c23ade69a960b6e2675 P 16440e365430405fb10366ab70e2b429: Deleting consensus metadata
I20240503 02:12:57.021587 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet 537c249dd4a44c23ade69a960b6e2675 (table rebalance_test_table_4 [id=e0ea42a974fb456485b832e1a5cb54ef]) successfully deleted
I20240503 02:12:57.023450 779 tablet_bootstrap.cc:492] T 9be5395489a3488da62f4d5b66facf3d P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:12:57.025184 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet 537c249dd4a44c23ade69a960b6e2675 (table rebalance_test_table_4 [id=e0ea42a974fb456485b832e1a5cb54ef]) successfully deleted
I20240503 02:12:57.025667 659 tablet_replica.cc:330] T fa8e2df1f59e4b86a5c18067108e07bf P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:12:57.026463 779 tablet_bootstrap.cc:654] T 9be5395489a3488da62f4d5b66facf3d P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:57.026691 659 raft_consensus.cc:2229] T fa8e2df1f59e4b86a5c18067108e07bf P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:57.027539 659 raft_consensus.cc:2258] T fa8e2df1f59e4b86a5c18067108e07bf P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:57.027558 777 consensus_queue.cc:260] T 9be5395489a3488da62f4d5b66facf3d P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:57.028478 659 ts_tablet_manager.cc:1875] T fa8e2df1f59e4b86a5c18067108e07bf P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:57.029060 777 ts_tablet_manager.cc:1398] T 9be5395489a3488da62f4d5b66facf3d P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240503 02:12:57.029421 777 tablet_bootstrap.cc:492] T 00aedc4902a745d6a859cbbf99529a7f P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:12:57.032056 777 tablet_bootstrap.cc:654] T 00aedc4902a745d6a859cbbf99529a7f P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:57.040134 659 ts_tablet_manager.cc:1888] T fa8e2df1f59e4b86a5c18067108e07bf P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:57.040309 779 tablet_bootstrap.cc:492] T 9be5395489a3488da62f4d5b66facf3d P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:12:57.040428 659 log.cc:1196] T fa8e2df1f59e4b86a5c18067108e07bf P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/fa8e2df1f59e4b86a5c18067108e07bf
I20240503 02:12:57.040544 779 ts_tablet_manager.cc:1367] T 9be5395489a3488da62f4d5b66facf3d P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20240503 02:12:57.040971 659 ts_tablet_manager.cc:1909] T fa8e2df1f59e4b86a5c18067108e07bf P 16440e365430405fb10366ab70e2b429: Deleting consensus metadata
I20240503 02:12:57.041426 779 raft_consensus.cc:348] T 9be5395489a3488da62f4d5b66facf3d P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:57.041707 779 raft_consensus.cc:374] T 9be5395489a3488da62f4d5b66facf3d P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:57.041818 779 raft_consensus.cc:729] T 9be5395489a3488da62f4d5b66facf3d P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:12:57.042189 779 consensus_queue.cc:260] T 9be5395489a3488da62f4d5b66facf3d P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:57.042940 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet fa8e2df1f59e4b86a5c18067108e07bf (table rebalance_test_table_4 [id=e0ea42a974fb456485b832e1a5cb54ef]) successfully deleted
I20240503 02:12:57.043399 779 ts_tablet_manager.cc:1398] T 9be5395489a3488da62f4d5b66facf3d P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:57.039233 777 tablet_bootstrap.cc:492] T 00aedc4902a745d6a859cbbf99529a7f P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:12:57.043797 777 ts_tablet_manager.cc:1367] T 00aedc4902a745d6a859cbbf99529a7f P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.004s
I20240503 02:12:57.044478 777 raft_consensus.cc:348] T 00aedc4902a745d6a859cbbf99529a7f P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:57.044873 656 ts_tablet_manager.cc:1888] T fa8e2df1f59e4b86a5c18067108e07bf P 1f10f80f13c84e80b03eda7085a3b012: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:57.045130 656 log.cc:1196] T fa8e2df1f59e4b86a5c18067108e07bf P 1f10f80f13c84e80b03eda7085a3b012: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/wals/fa8e2df1f59e4b86a5c18067108e07bf
I20240503 02:12:57.046470 777 raft_consensus.cc:374] T 00aedc4902a745d6a859cbbf99529a7f P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:57.046777 777 raft_consensus.cc:729] T 00aedc4902a745d6a859cbbf99529a7f P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:12:57.047250 777 consensus_queue.cc:260] T 00aedc4902a745d6a859cbbf99529a7f P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:57.045940 656 ts_tablet_manager.cc:1909] T fa8e2df1f59e4b86a5c18067108e07bf P 1f10f80f13c84e80b03eda7085a3b012: Deleting consensus metadata
I20240503 02:12:57.048416 29719 catalog_manager.cc:4830] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): tablet fa8e2df1f59e4b86a5c18067108e07bf (table rebalance_test_table_4 [id=e0ea42a974fb456485b832e1a5cb54ef]) successfully deleted
I20240503 02:12:57.049466 777 ts_tablet_manager.cc:1398] T 00aedc4902a745d6a859cbbf99529a7f P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 02:12:57.103181 693 raft_consensus.cc:482] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:57.103421 693 raft_consensus.cc:504] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:57.104373 693 leader_election.cc:288] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:12:57.104820 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9be5395489a3488da62f4d5b66facf3d" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:12:57.105098 32641 raft_consensus.cc:2454] T 9be5395489a3488da62f4d5b66facf3d P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 0.
I20240503 02:12:57.115742 502 leader_election.cc:302] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:12:57.116163 693 raft_consensus.cc:2781] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:57.116269 693 raft_consensus.cc:482] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:57.116375 693 raft_consensus.cc:3037] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:57.119321 693 raft_consensus.cc:504] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:57.119594 709 raft_consensus.cc:482] T 5ccde7688c4e49b5b9d89d6e048c308e P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:57.119776 709 raft_consensus.cc:504] T 5ccde7688c4e49b5b9d89d6e048c308e P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:57.119939 693 leader_election.cc:288] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 election: Requested vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:12:57.120380 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9be5395489a3488da62f4d5b66facf3d" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d"
I20240503 02:12:57.120585 709 leader_election.cc:288] T 5ccde7688c4e49b5b9d89d6e048c308e P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:12:57.120613 32641 raft_consensus.cc:3037] T 9be5395489a3488da62f4d5b66facf3d P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:57.122272 29734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:12:57.123276 32641 raft_consensus.cc:2454] T 9be5395489a3488da62f4d5b66facf3d P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 1.
I20240503 02:12:57.123885 502 leader_election.cc:302] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:12:57.124694 693 raft_consensus.cc:2781] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:57.126255 693 raft_consensus.cc:686] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe [term 1 LEADER]: Becoming Leader. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Running, Role: LEADER
I20240503 02:12:57.126859 693 consensus_queue.cc:237] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:57.133178 794 tablet.cc:1722] T 7c1b368d5f8d46f1a5e56a3ff746e77b P a9094240860e43bfb129193f3bfd5be6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:12:57.133720 794 tablet.cc:1620] T 7c1b368d5f8d46f1a5e56a3ff746e77b P a9094240860e43bfb129193f3bfd5be6: MemRowSet was empty: no flush needed.
I20240503 02:12:57.135030 795 tablet.cc:1722] T 7c1b368d5f8d46f1a5e56a3ff746e77b P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:12:57.135552 795 tablet.cc:1620] T 7c1b368d5f8d46f1a5e56a3ff746e77b P fbbc2be6382e47908ecb7c68c367bd26: MemRowSet was empty: no flush needed.
I20240503 02:12:57.142148 800 tablet.cc:1722] T 7c1b368d5f8d46f1a5e56a3ff746e77b P 1f10f80f13c84e80b03eda7085a3b012: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:12:57.142673 800 tablet.cc:1620] T 7c1b368d5f8d46f1a5e56a3ff746e77b P 1f10f80f13c84e80b03eda7085a3b012: MemRowSet was empty: no flush needed.
I20240503 02:12:57.143177 692 raft_consensus.cc:482] T 5ccde7688c4e49b5b9d89d6e048c308e P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:57.143347 692 raft_consensus.cc:504] T 5ccde7688c4e49b5b9d89d6e048c308e P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:57.144157 692 leader_election.cc:288] T 5ccde7688c4e49b5b9d89d6e048c308e P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:12:57.148270 29730 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=17fce000d51e45689be7e25cb26c8067] alter complete (version 2)
I20240503 02:12:57.149361 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ccde7688c4e49b5b9d89d6e048c308e" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:12:57.150233 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ccde7688c4e49b5b9d89d6e048c308e" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:12:57.150246 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9be5395489a3488da62f4d5b66facf3d" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:12:57.150461 437 raft_consensus.cc:2454] T 5ccde7688c4e49b5b9d89d6e048c308e P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:12:57.150523 307 raft_consensus.cc:2454] T 9be5395489a3488da62f4d5b66facf3d P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 0.
I20240503 02:12:57.150991 32641 raft_consensus.cc:2454] T 5ccde7688c4e49b5b9d89d6e048c308e P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 0.
I20240503 02:12:57.150985 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9be5395489a3488da62f4d5b66facf3d" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:12:57.151240 307 raft_consensus.cc:3037] T 9be5395489a3488da62f4d5b66facf3d P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:57.151549 371 leader_election.cc:302] T 5ccde7688c4e49b5b9d89d6e048c308e P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:12:57.151918 692 raft_consensus.cc:2781] T 5ccde7688c4e49b5b9d89d6e048c308e P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:57.152015 692 raft_consensus.cc:482] T 5ccde7688c4e49b5b9d89d6e048c308e P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:57.152112 692 raft_consensus.cc:3037] T 5ccde7688c4e49b5b9d89d6e048c308e P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:57.154596 307 raft_consensus.cc:2454] T 9be5395489a3488da62f4d5b66facf3d P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 1.
I20240503 02:12:57.154973 692 raft_consensus.cc:504] T 5ccde7688c4e49b5b9d89d6e048c308e P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:57.156422 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ccde7688c4e49b5b9d89d6e048c308e" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:12:57.156458 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ccde7688c4e49b5b9d89d6e048c308e" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d"
I20240503 02:12:57.156649 32640 raft_consensus.cc:3037] T 5ccde7688c4e49b5b9d89d6e048c308e P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:57.156697 307 raft_consensus.cc:2454] T 5ccde7688c4e49b5b9d89d6e048c308e P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 0.
I20240503 02:12:57.156760 32709 leader_election.cc:302] T 5ccde7688c4e49b5b9d89d6e048c308e P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:12:57.157203 709 raft_consensus.cc:2781] T 5ccde7688c4e49b5b9d89d6e048c308e P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:57.157287 709 raft_consensus.cc:482] T 5ccde7688c4e49b5b9d89d6e048c308e P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:57.157382 709 raft_consensus.cc:3037] T 5ccde7688c4e49b5b9d89d6e048c308e P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:57.158676 32640 raft_consensus.cc:2454] T 5ccde7688c4e49b5b9d89d6e048c308e P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 1.
I20240503 02:12:57.159086 371 leader_election.cc:302] T 5ccde7688c4e49b5b9d89d6e048c308e P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:12:57.159078 306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ccde7688c4e49b5b9d89d6e048c308e" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:12:57.160001 709 raft_consensus.cc:504] T 5ccde7688c4e49b5b9d89d6e048c308e P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:57.160416 306 raft_consensus.cc:2379] T 5ccde7688c4e49b5b9d89d6e048c308e P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fbbc2be6382e47908ecb7c68c367bd26 in current term 1: Already voted for candidate 8a666e798fe8477d8dd1e51b385d5ead in this term.
I20240503 02:12:57.160678 709 leader_election.cc:288] T 5ccde7688c4e49b5b9d89d6e048c308e P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Requested vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:12:57.162178 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ccde7688c4e49b5b9d89d6e048c308e" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d"
I20240503 02:12:57.162426 32640 raft_consensus.cc:2379] T 5ccde7688c4e49b5b9d89d6e048c308e P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a666e798fe8477d8dd1e51b385d5ead in current term 1: Already voted for candidate fbbc2be6382e47908ecb7c68c367bd26 in this term.
I20240503 02:12:57.162981 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ccde7688c4e49b5b9d89d6e048c308e" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:12:57.165483 32641 raft_consensus.cc:2379] T 5ccde7688c4e49b5b9d89d6e048c308e P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8a666e798fe8477d8dd1e51b385d5ead in current term 1: Already voted for candidate fbbc2be6382e47908ecb7c68c367bd26 in this term.
I20240503 02:12:57.166283 762 raft_consensus.cc:2781] T 5ccde7688c4e49b5b9d89d6e048c308e P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:57.166703 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ccde7688c4e49b5b9d89d6e048c308e" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:12:57.167078 762 raft_consensus.cc:686] T 5ccde7688c4e49b5b9d89d6e048c308e P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Becoming Leader. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Running, Role: LEADER
I20240503 02:12:57.167382 762 consensus_queue.cc:237] T 5ccde7688c4e49b5b9d89d6e048c308e P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:57.169371 692 leader_election.cc:288] T 5ccde7688c4e49b5b9d89d6e048c308e P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Requested vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:12:57.170578 29730 catalog_manager.cc:5484] T 5ccde7688c4e49b5b9d89d6e048c308e P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7). New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:12:57.171056 32709 leader_election.cc:302] T 5ccde7688c4e49b5b9d89d6e048c308e P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead; no voters: 9a4d92bc25374fb3960fde7d10be309d, fbbc2be6382e47908ecb7c68c367bd26
I20240503 02:12:57.171643 709 raft_consensus.cc:482] T 00aedc4902a745d6a859cbbf99529a7f P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:57.171877 709 raft_consensus.cc:504] T 00aedc4902a745d6a859cbbf99529a7f P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:57.172839 709 leader_election.cc:288] T 00aedc4902a745d6a859cbbf99529a7f P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:12:57.175961 643 raft_consensus.cc:2726] T 5ccde7688c4e49b5b9d89d6e048c308e P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 02:12:57.176571 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00aedc4902a745d6a859cbbf99529a7f" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:12:57.176841 32641 raft_consensus.cc:2454] T 00aedc4902a745d6a859cbbf99529a7f P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:12:57.177232 32709 leader_election.cc:302] T 00aedc4902a745d6a859cbbf99529a7f P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, 9a4d92bc25374fb3960fde7d10be309d; no voters:
I20240503 02:12:57.178260 643 raft_consensus.cc:2781] T 00aedc4902a745d6a859cbbf99529a7f P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:57.178381 643 raft_consensus.cc:482] T 00aedc4902a745d6a859cbbf99529a7f P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:57.178506 643 raft_consensus.cc:3037] T 00aedc4902a745d6a859cbbf99529a7f P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:57.179200 29730 catalog_manager.cc:5484] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: term changed from 0 to 1, leader changed from <none> to f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8). New cstate: current_term: 1 leader_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:12:57.181429 643 raft_consensus.cc:504] T 00aedc4902a745d6a859cbbf99529a7f P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:57.182232 643 leader_election.cc:288] T 00aedc4902a745d6a859cbbf99529a7f P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Requested vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:12:57.182987 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00aedc4902a745d6a859cbbf99529a7f" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d"
I20240503 02:12:57.183262 32641 raft_consensus.cc:3037] T 00aedc4902a745d6a859cbbf99529a7f P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:57.185463 32641 raft_consensus.cc:2454] T 00aedc4902a745d6a859cbbf99529a7f P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:12:57.185961 32709 leader_election.cc:302] T 00aedc4902a745d6a859cbbf99529a7f P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, 9a4d92bc25374fb3960fde7d10be309d; no voters:
I20240503 02:12:57.186272 709 raft_consensus.cc:2781] T 00aedc4902a745d6a859cbbf99529a7f P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:57.186442 709 raft_consensus.cc:686] T 00aedc4902a745d6a859cbbf99529a7f P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Becoming Leader. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Running, Role: LEADER
I20240503 02:12:57.186708 709 consensus_queue.cc:237] T 00aedc4902a745d6a859cbbf99529a7f P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:57.192473 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00aedc4902a745d6a859cbbf99529a7f" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:12:57.192764 30266 raft_consensus.cc:2454] T 00aedc4902a745d6a859cbbf99529a7f P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:12:57.193006 29730 catalog_manager.cc:5484] T 00aedc4902a745d6a859cbbf99529a7f P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6). New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:12:57.193331 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00aedc4902a745d6a859cbbf99529a7f" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:12:57.193547 30264 raft_consensus.cc:3037] T 00aedc4902a745d6a859cbbf99529a7f P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:57.199396 30264 raft_consensus.cc:2454] T 00aedc4902a745d6a859cbbf99529a7f P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:12:57.256582 29725 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:12:57.275461 794 tablet.cc:1722] T 7c1b368d5f8d46f1a5e56a3ff746e77b P a9094240860e43bfb129193f3bfd5be6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:12:57.285056 795 tablet.cc:1722] T 7c1b368d5f8d46f1a5e56a3ff746e77b P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:12:57.285742 800 tablet.cc:1722] T 7c1b368d5f8d46f1a5e56a3ff746e77b P 1f10f80f13c84e80b03eda7085a3b012: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:12:57.292886 29727 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table_new_name [id=17fce000d51e45689be7e25cb26c8067] alter complete (version 3)
I20240503 02:12:57.353394 29986 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:12:57.379323 30114 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:12:57.396438 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:12:57.397373 29730 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:12:57.397640 29730 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:12:57.398770 32754 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:57.401059 29730 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet 7c1b368d5f8d46f1a5e56a3ff746e77b
I20240503 02:12:57.401125 29853 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:12:57.402216 30244 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c1b368d5f8d46f1a5e56a3ff746e77b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:57 UTC) from {username='slave'} at 127.0.0.1:49856
I20240503 02:12:57.402531 656 tablet_replica.cc:330] T 7c1b368d5f8d46f1a5e56a3ff746e77b P 1f10f80f13c84e80b03eda7085a3b012: stopping tablet replica
I20240503 02:12:57.402750 656 raft_consensus.cc:2229] T 7c1b368d5f8d46f1a5e56a3ff746e77b P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:57.402959 656 raft_consensus.cc:2258] T 7c1b368d5f8d46f1a5e56a3ff746e77b P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:57.404259 656 ts_tablet_manager.cc:1875] T 7c1b368d5f8d46f1a5e56a3ff746e77b P 1f10f80f13c84e80b03eda7085a3b012: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:57.405845 30114 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c1b368d5f8d46f1a5e56a3ff746e77b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:57 UTC) from {username='slave'} at 127.0.0.1:44812
I20240503 02:12:57.406148 796 tablet_replica.cc:330] T 7c1b368d5f8d46f1a5e56a3ff746e77b P a9094240860e43bfb129193f3bfd5be6: stopping tablet replica
I20240503 02:12:57.406334 796 raft_consensus.cc:2229] T 7c1b368d5f8d46f1a5e56a3ff746e77b P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:57.406703 796 raft_consensus.cc:2258] T 7c1b368d5f8d46f1a5e56a3ff746e77b P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:57.408728 796 ts_tablet_manager.cc:1875] T 7c1b368d5f8d46f1a5e56a3ff746e77b P a9094240860e43bfb129193f3bfd5be6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:57.411298 656 ts_tablet_manager.cc:1888] T 7c1b368d5f8d46f1a5e56a3ff746e77b P 1f10f80f13c84e80b03eda7085a3b012: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:57.411583 656 log.cc:1196] T 7c1b368d5f8d46f1a5e56a3ff746e77b P 1f10f80f13c84e80b03eda7085a3b012: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/wals/7c1b368d5f8d46f1a5e56a3ff746e77b
I20240503 02:12:57.412207 656 ts_tablet_manager.cc:1909] T 7c1b368d5f8d46f1a5e56a3ff746e77b P 1f10f80f13c84e80b03eda7085a3b012: Deleting consensus metadata
I20240503 02:12:57.414508 29719 catalog_manager.cc:4830] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): tablet 7c1b368d5f8d46f1a5e56a3ff746e77b (table rebalancer_dynamic_table_new_name [id=17fce000d51e45689be7e25cb26c8067]) successfully deleted
I20240503 02:12:57.425525 796 ts_tablet_manager.cc:1888] T 7c1b368d5f8d46f1a5e56a3ff746e77b P a9094240860e43bfb129193f3bfd5be6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:57.425671 796 log.cc:1196] T 7c1b368d5f8d46f1a5e56a3ff746e77b P a9094240860e43bfb129193f3bfd5be6: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/wals/7c1b368d5f8d46f1a5e56a3ff746e77b
I20240503 02:12:57.425849 417 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c1b368d5f8d46f1a5e56a3ff746e77b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:57 UTC) from {username='slave'} at 127.0.0.1:50814
I20240503 02:12:57.426108 796 ts_tablet_manager.cc:1909] T 7c1b368d5f8d46f1a5e56a3ff746e77b P a9094240860e43bfb129193f3bfd5be6: Deleting consensus metadata
I20240503 02:12:57.433059 546 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:57.444638 29718 catalog_manager.cc:4830] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): tablet 7c1b368d5f8d46f1a5e56a3ff746e77b (table rebalancer_dynamic_table_new_name [id=17fce000d51e45689be7e25cb26c8067]) successfully deleted
I20240503 02:12:57.457542 867 tablet_replica.cc:330] T 7c1b368d5f8d46f1a5e56a3ff746e77b P fbbc2be6382e47908ecb7c68c367bd26: stopping tablet replica
I20240503 02:12:57.457805 867 raft_consensus.cc:2229] T 7c1b368d5f8d46f1a5e56a3ff746e77b P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:57.458070 867 raft_consensus.cc:2258] T 7c1b368d5f8d46f1a5e56a3ff746e77b P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:57.465700 867 ts_tablet_manager.cc:1875] T 7c1b368d5f8d46f1a5e56a3ff746e77b P fbbc2be6382e47908ecb7c68c367bd26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:57.472231 867 ts_tablet_manager.cc:1888] T 7c1b368d5f8d46f1a5e56a3ff746e77b P fbbc2be6382e47908ecb7c68c367bd26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:57.472364 867 log.cc:1196] T 7c1b368d5f8d46f1a5e56a3ff746e77b P fbbc2be6382e47908ecb7c68c367bd26: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal/wals/7c1b368d5f8d46f1a5e56a3ff746e77b
I20240503 02:12:57.472680 29730 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240503 02:12:57.472929 29730 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240503 02:12:57.477397 29730 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet 636257ce6f0f43cdb764aae55a9e764e
I20240503 02:12:57.477943 29730 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet d4a5af9181d0429a8a4f4d734c59e2b6
I20240503 02:12:57.478937 30114 tablet_service.cc:1514] Processing DeleteTablet for tablet d4a5af9181d0429a8a4f4d734c59e2b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:57 UTC) from {username='slave'} at 127.0.0.1:44812
I20240503 02:12:57.479296 796 tablet_replica.cc:330] T d4a5af9181d0429a8a4f4d734c59e2b6 P a9094240860e43bfb129193f3bfd5be6: stopping tablet replica
I20240503 02:12:57.479308 29853 tablet_service.cc:1514] Processing DeleteTablet for tablet 636257ce6f0f43cdb764aae55a9e764e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:57 UTC) from {username='slave'} at 127.0.0.1:58744
I20240503 02:12:57.479591 659 tablet_replica.cc:330] T 636257ce6f0f43cdb764aae55a9e764e P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:12:57.479615 29986 tablet_service.cc:1514] Processing DeleteTablet for tablet 636257ce6f0f43cdb764aae55a9e764e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:57 UTC) from {username='slave'} at 127.0.0.1:52158
I20240503 02:12:57.479655 796 raft_consensus.cc:2229] T d4a5af9181d0429a8a4f4d734c59e2b6 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:57.479853 659 raft_consensus.cc:2229] T 636257ce6f0f43cdb764aae55a9e764e P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:57.479912 657 tablet_replica.cc:330] T 636257ce6f0f43cdb764aae55a9e764e P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:12:57.480115 659 raft_consensus.cc:2258] T 636257ce6f0f43cdb764aae55a9e764e P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:57.480126 657 raft_consensus.cc:2229] T 636257ce6f0f43cdb764aae55a9e764e P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:57.480353 657 raft_consensus.cc:2258] T 636257ce6f0f43cdb764aae55a9e764e P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:57.480937 659 ts_tablet_manager.cc:1875] T 636257ce6f0f43cdb764aae55a9e764e P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:57.481082 657 ts_tablet_manager.cc:1875] T 636257ce6f0f43cdb764aae55a9e764e P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:57.482182 417 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:57.483690 32621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:57.483891 30244 tablet_service.cc:1514] Processing DeleteTablet for tablet 636257ce6f0f43cdb764aae55a9e764e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:57 UTC) from {username='slave'} at 127.0.0.1:49856
I20240503 02:12:57.484225 656 tablet_replica.cc:330] T 636257ce6f0f43cdb764aae55a9e764e P 1f10f80f13c84e80b03eda7085a3b012: stopping tablet replica
I20240503 02:12:57.484424 656 raft_consensus.cc:2229] T 636257ce6f0f43cdb764aae55a9e764e P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:57.484818 656 raft_consensus.cc:2258] T 636257ce6f0f43cdb764aae55a9e764e P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:57.485574 656 ts_tablet_manager.cc:1875] T 636257ce6f0f43cdb764aae55a9e764e P 1f10f80f13c84e80b03eda7085a3b012: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:57.485821 29853 tablet_service.cc:1514] Processing DeleteTablet for tablet d4a5af9181d0429a8a4f4d734c59e2b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:57 UTC) from {username='slave'} at 127.0.0.1:58744
I20240503 02:12:57.486428 29730 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet 8a9e3025d1764ce1aa48071ac3e5d902
I20240503 02:12:57.487149 29986 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a9e3025d1764ce1aa48071ac3e5d902 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:57 UTC) from {username='slave'} at 127.0.0.1:52158
I20240503 02:12:57.487844 30245 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a9e3025d1764ce1aa48071ac3e5d902 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:57 UTC) from {username='slave'} at 127.0.0.1:49856
I20240503 02:12:57.489207 30244 tablet_service.cc:1514] Processing DeleteTablet for tablet d4a5af9181d0429a8a4f4d734c59e2b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:57 UTC) from {username='slave'} at 127.0.0.1:49856
I20240503 02:12:57.489298 29853 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a9e3025d1764ce1aa48071ac3e5d902 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:57 UTC) from {username='slave'} at 127.0.0.1:58744
I20240503 02:12:57.491715 657 ts_tablet_manager.cc:1888] T 636257ce6f0f43cdb764aae55a9e764e P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:57.491878 657 log.cc:1196] T 636257ce6f0f43cdb764aae55a9e764e P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/636257ce6f0f43cdb764aae55a9e764e
I20240503 02:12:57.491866 796 raft_consensus.cc:2258] T d4a5af9181d0429a8a4f4d734c59e2b6 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:57.492702 657 ts_tablet_manager.cc:1909] T 636257ce6f0f43cdb764aae55a9e764e P 155735c7f8b14997ac4c770aee2a9401: Deleting consensus metadata
I20240503 02:12:57.493109 796 ts_tablet_manager.cc:1875] T d4a5af9181d0429a8a4f4d734c59e2b6 P a9094240860e43bfb129193f3bfd5be6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:57.493804 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet 636257ce6f0f43cdb764aae55a9e764e (table rebalance_test_table_6 [id=fdf8a5db2b9b450b8db9d7a71433d20b]) successfully deleted
I20240503 02:12:57.494333 656 ts_tablet_manager.cc:1888] T 636257ce6f0f43cdb764aae55a9e764e P 1f10f80f13c84e80b03eda7085a3b012: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:57.494462 656 log.cc:1196] T 636257ce6f0f43cdb764aae55a9e764e P 1f10f80f13c84e80b03eda7085a3b012: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/wals/636257ce6f0f43cdb764aae55a9e764e
I20240503 02:12:57.494920 656 ts_tablet_manager.cc:1909] T 636257ce6f0f43cdb764aae55a9e764e P 1f10f80f13c84e80b03eda7085a3b012: Deleting consensus metadata
I20240503 02:12:57.497514 659 ts_tablet_manager.cc:1888] T 636257ce6f0f43cdb764aae55a9e764e P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:57.497767 659 log.cc:1196] T 636257ce6f0f43cdb764aae55a9e764e P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/636257ce6f0f43cdb764aae55a9e764e
I20240503 02:12:57.498201 659 ts_tablet_manager.cc:1909] T 636257ce6f0f43cdb764aae55a9e764e P 16440e365430405fb10366ab70e2b429: Deleting consensus metadata
I20240503 02:12:57.499116 659 tablet_replica.cc:330] T d4a5af9181d0429a8a4f4d734c59e2b6 P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:12:57.499246 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet 636257ce6f0f43cdb764aae55a9e764e (table rebalance_test_table_6 [id=fdf8a5db2b9b450b8db9d7a71433d20b]) successfully deleted
I20240503 02:12:57.499312 659 raft_consensus.cc:2229] T d4a5af9181d0429a8a4f4d734c59e2b6 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:57.498975 657 tablet_replica.cc:330] T 8a9e3025d1764ce1aa48071ac3e5d902 P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:12:57.499642 659 raft_consensus.cc:2258] T d4a5af9181d0429a8a4f4d734c59e2b6 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:57.499656 657 raft_consensus.cc:2229] T 8a9e3025d1764ce1aa48071ac3e5d902 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:57.499997 657 raft_consensus.cc:2258] T 8a9e3025d1764ce1aa48071ac3e5d902 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:57.500365 659 ts_tablet_manager.cc:1875] T d4a5af9181d0429a8a4f4d734c59e2b6 P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:57.500700 657 ts_tablet_manager.cc:1875] T 8a9e3025d1764ce1aa48071ac3e5d902 P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:57.503682 656 tablet_replica.cc:330] T 8a9e3025d1764ce1aa48071ac3e5d902 P 1f10f80f13c84e80b03eda7085a3b012: stopping tablet replica
I20240503 02:12:57.503934 656 raft_consensus.cc:2229] T 8a9e3025d1764ce1aa48071ac3e5d902 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:57.503705 29730 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:12:57.504045 29719 catalog_manager.cc:4830] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): tablet 636257ce6f0f43cdb764aae55a9e764e (table rebalance_test_table_6 [id=fdf8a5db2b9b450b8db9d7a71433d20b]) successfully deleted
I20240503 02:12:57.504225 656 raft_consensus.cc:2258] T 8a9e3025d1764ce1aa48071ac3e5d902 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:57.504954 656 ts_tablet_manager.cc:1875] T 8a9e3025d1764ce1aa48071ac3e5d902 P 1f10f80f13c84e80b03eda7085a3b012: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:57.512583 546 tablet_service.cc:1467] Processing CreateTablet for tablet c50aed6222af431abf62a57380647401 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b77a8bed065f4ea798fca0b608e4a1f1]), partition=
I20240503 02:12:57.512780 32754 tablet_service.cc:1467] Processing CreateTablet for tablet c50aed6222af431abf62a57380647401 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b77a8bed065f4ea798fca0b608e4a1f1]), partition=
I20240503 02:12:57.513001 546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c50aed6222af431abf62a57380647401. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:57.513187 32754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c50aed6222af431abf62a57380647401. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:57.516983 417 tablet_service.cc:1467] Processing CreateTablet for tablet c50aed6222af431abf62a57380647401 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b77a8bed065f4ea798fca0b608e4a1f1]), partition=
I20240503 02:12:57.517354 417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c50aed6222af431abf62a57380647401. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:57.526916 779 tablet_bootstrap.cc:492] T c50aed6222af431abf62a57380647401 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:12:57.528491 779 tablet_bootstrap.cc:654] T c50aed6222af431abf62a57380647401 P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:57.535282 868 tablet_bootstrap.cc:492] T c50aed6222af431abf62a57380647401 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:12:57.536876 868 tablet_bootstrap.cc:654] T c50aed6222af431abf62a57380647401 P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:57.537146 659 ts_tablet_manager.cc:1888] T d4a5af9181d0429a8a4f4d734c59e2b6 P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:57.537297 659 log.cc:1196] T d4a5af9181d0429a8a4f4d734c59e2b6 P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/d4a5af9181d0429a8a4f4d734c59e2b6
I20240503 02:12:57.537794 659 ts_tablet_manager.cc:1909] T d4a5af9181d0429a8a4f4d734c59e2b6 P 16440e365430405fb10366ab70e2b429: Deleting consensus metadata
I20240503 02:12:57.538002 867 ts_tablet_manager.cc:1909] T 7c1b368d5f8d46f1a5e56a3ff746e77b P fbbc2be6382e47908ecb7c68c367bd26: Deleting consensus metadata
I20240503 02:12:57.539052 29719 catalog_manager.cc:4830] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): tablet 7c1b368d5f8d46f1a5e56a3ff746e77b (table rebalancer_dynamic_table_new_name [id=17fce000d51e45689be7e25cb26c8067]) successfully deleted
I20240503 02:12:57.539062 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet d4a5af9181d0429a8a4f4d734c59e2b6 (table rebalance_test_table_6 [id=fdf8a5db2b9b450b8db9d7a71433d20b]) successfully deleted
I20240503 02:12:57.541507 869 tablet_bootstrap.cc:492] T c50aed6222af431abf62a57380647401 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:12:57.543144 656 ts_tablet_manager.cc:1888] T 8a9e3025d1764ce1aa48071ac3e5d902 P 1f10f80f13c84e80b03eda7085a3b012: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:57.543216 869 tablet_bootstrap.cc:654] T c50aed6222af431abf62a57380647401 P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:57.543316 656 log.cc:1196] T 8a9e3025d1764ce1aa48071ac3e5d902 P 1f10f80f13c84e80b03eda7085a3b012: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/wals/8a9e3025d1764ce1aa48071ac3e5d902
I20240503 02:12:57.543802 656 ts_tablet_manager.cc:1909] T 8a9e3025d1764ce1aa48071ac3e5d902 P 1f10f80f13c84e80b03eda7085a3b012: Deleting consensus metadata
I20240503 02:12:57.544567 779 tablet_bootstrap.cc:492] T c50aed6222af431abf62a57380647401 P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:12:57.544817 779 ts_tablet_manager.cc:1367] T c50aed6222af431abf62a57380647401 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.000s
I20240503 02:12:57.545679 779 raft_consensus.cc:348] T c50aed6222af431abf62a57380647401 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:57.545944 29719 catalog_manager.cc:4830] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): tablet 8a9e3025d1764ce1aa48071ac3e5d902 (table rebalance_test_table_6 [id=fdf8a5db2b9b450b8db9d7a71433d20b]) successfully deleted
I20240503 02:12:57.546082 779 raft_consensus.cc:374] T c50aed6222af431abf62a57380647401 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:57.546204 779 raft_consensus.cc:729] T c50aed6222af431abf62a57380647401 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:12:57.546604 779 consensus_queue.cc:260] T c50aed6222af431abf62a57380647401 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:57.548357 656 tablet_replica.cc:330] T d4a5af9181d0429a8a4f4d734c59e2b6 P 1f10f80f13c84e80b03eda7085a3b012: stopping tablet replica
I20240503 02:12:57.549050 779 ts_tablet_manager.cc:1398] T c50aed6222af431abf62a57380647401 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:12:57.551734 693 consensus_queue.cc:1035] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:57.556655 692 consensus_queue.cc:1035] T 5ccde7688c4e49b5b9d89d6e048c308e P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:57.557463 657 ts_tablet_manager.cc:1888] T 8a9e3025d1764ce1aa48071ac3e5d902 P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:57.557615 657 log.cc:1196] T 8a9e3025d1764ce1aa48071ac3e5d902 P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/8a9e3025d1764ce1aa48071ac3e5d902
I20240503 02:12:57.558140 657 ts_tablet_manager.cc:1909] T 8a9e3025d1764ce1aa48071ac3e5d902 P 155735c7f8b14997ac4c770aee2a9401: Deleting consensus metadata
I20240503 02:12:57.559128 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet 8a9e3025d1764ce1aa48071ac3e5d902 (table rebalance_test_table_6 [id=fdf8a5db2b9b450b8db9d7a71433d20b]) successfully deleted
I20240503 02:12:57.570591 869 tablet_bootstrap.cc:492] T c50aed6222af431abf62a57380647401 P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:12:57.571152 869 ts_tablet_manager.cc:1367] T c50aed6222af431abf62a57380647401 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.006s
I20240503 02:12:57.572444 656 raft_consensus.cc:2229] T d4a5af9181d0429a8a4f4d734c59e2b6 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:57.572809 656 raft_consensus.cc:2258] T d4a5af9181d0429a8a4f4d734c59e2b6 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:57.573678 796 ts_tablet_manager.cc:1888] T d4a5af9181d0429a8a4f4d734c59e2b6 P a9094240860e43bfb129193f3bfd5be6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:57.573854 796 log.cc:1196] T d4a5af9181d0429a8a4f4d734c59e2b6 P a9094240860e43bfb129193f3bfd5be6: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/wals/d4a5af9181d0429a8a4f4d734c59e2b6
I20240503 02:12:57.572813 869 raft_consensus.cc:348] T c50aed6222af431abf62a57380647401 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:57.575587 869 raft_consensus.cc:374] T c50aed6222af431abf62a57380647401 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:57.575739 869 raft_consensus.cc:729] T c50aed6222af431abf62a57380647401 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:12:57.576087 869 consensus_queue.cc:260] T c50aed6222af431abf62a57380647401 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:57.576551 656 ts_tablet_manager.cc:1875] T d4a5af9181d0429a8a4f4d734c59e2b6 P 1f10f80f13c84e80b03eda7085a3b012: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:57.577430 868 tablet_bootstrap.cc:492] T c50aed6222af431abf62a57380647401 P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:12:57.577630 868 ts_tablet_manager.cc:1367] T c50aed6222af431abf62a57380647401 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.042s user 0.001s sys 0.005s
I20240503 02:12:57.578465 868 raft_consensus.cc:348] T c50aed6222af431abf62a57380647401 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:57.578740 868 raft_consensus.cc:374] T c50aed6222af431abf62a57380647401 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:57.578913 868 raft_consensus.cc:729] T c50aed6222af431abf62a57380647401 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:12:57.579218 868 consensus_queue.cc:260] T c50aed6222af431abf62a57380647401 P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:57.579373 796 ts_tablet_manager.cc:1909] T d4a5af9181d0429a8a4f4d734c59e2b6 P a9094240860e43bfb129193f3bfd5be6: Deleting consensus metadata
I20240503 02:12:57.583230 659 tablet_replica.cc:330] T 8a9e3025d1764ce1aa48071ac3e5d902 P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:12:57.583495 659 raft_consensus.cc:2229] T 8a9e3025d1764ce1aa48071ac3e5d902 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:57.583777 659 raft_consensus.cc:2258] T 8a9e3025d1764ce1aa48071ac3e5d902 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:57.583803 868 ts_tablet_manager.cc:1398] T c50aed6222af431abf62a57380647401 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:12:57.584641 659 ts_tablet_manager.cc:1875] T 8a9e3025d1764ce1aa48071ac3e5d902 P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:57.585956 693 raft_consensus.cc:482] T c50aed6222af431abf62a57380647401 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:57.586103 693 raft_consensus.cc:504] T c50aed6222af431abf62a57380647401 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:57.586760 693 leader_election.cc:288] T c50aed6222af431abf62a57380647401 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:12:57.587857 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c50aed6222af431abf62a57380647401" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:12:57.588171 305 raft_consensus.cc:2454] T c50aed6222af431abf62a57380647401 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 0.
I20240503 02:12:57.588891 29718 catalog_manager.cc:4830] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): tablet d4a5af9181d0429a8a4f4d734c59e2b6 (table rebalance_test_table_6 [id=fdf8a5db2b9b450b8db9d7a71433d20b]) successfully deleted
I20240503 02:12:57.588907 502 leader_election.cc:302] T c50aed6222af431abf62a57380647401 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:12:57.589663 693 raft_consensus.cc:2781] T c50aed6222af431abf62a57380647401 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:57.589771 693 raft_consensus.cc:482] T c50aed6222af431abf62a57380647401 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:57.589866 693 raft_consensus.cc:3037] T c50aed6222af431abf62a57380647401 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:57.592490 693 raft_consensus.cc:504] T c50aed6222af431abf62a57380647401 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:57.593935 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c50aed6222af431abf62a57380647401" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:12:57.594141 305 raft_consensus.cc:3037] T c50aed6222af431abf62a57380647401 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:57.596998 305 raft_consensus.cc:2454] T c50aed6222af431abf62a57380647401 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 1.
I20240503 02:12:57.600195 502 leader_election.cc:302] T c50aed6222af431abf62a57380647401 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:12:57.603060 659 ts_tablet_manager.cc:1888] T 8a9e3025d1764ce1aa48071ac3e5d902 P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:57.603215 659 log.cc:1196] T 8a9e3025d1764ce1aa48071ac3e5d902 P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/8a9e3025d1764ce1aa48071ac3e5d902
I20240503 02:12:57.603621 659 ts_tablet_manager.cc:1909] T 8a9e3025d1764ce1aa48071ac3e5d902 P 16440e365430405fb10366ab70e2b429: Deleting consensus metadata
I20240503 02:12:57.604743 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet 8a9e3025d1764ce1aa48071ac3e5d902 (table rebalance_test_table_6 [id=fdf8a5db2b9b450b8db9d7a71433d20b]) successfully deleted
I20240503 02:12:57.607036 693 leader_election.cc:288] T c50aed6222af431abf62a57380647401 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 election: Requested vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:12:57.609073 693 raft_consensus.cc:2781] T c50aed6222af431abf62a57380647401 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:57.609279 693 raft_consensus.cc:686] T c50aed6222af431abf62a57380647401 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 LEADER]: Becoming Leader. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Running, Role: LEADER
I20240503 02:12:57.609555 693 consensus_queue.cc:237] T c50aed6222af431abf62a57380647401 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:57.616495 29730 catalog_manager.cc:5484] T c50aed6222af431abf62a57380647401 P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: term changed from 0 to 1, leader changed from <none> to f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8). New cstate: current_term: 1 leader_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:12:57.618644 656 ts_tablet_manager.cc:1888] T d4a5af9181d0429a8a4f4d734c59e2b6 P 1f10f80f13c84e80b03eda7085a3b012: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:57.618809 656 log.cc:1196] T d4a5af9181d0429a8a4f4d734c59e2b6 P 1f10f80f13c84e80b03eda7085a3b012: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/wals/d4a5af9181d0429a8a4f4d734c59e2b6
I20240503 02:12:57.619215 656 ts_tablet_manager.cc:1909] T d4a5af9181d0429a8a4f4d734c59e2b6 P 1f10f80f13c84e80b03eda7085a3b012: Deleting consensus metadata
I20240503 02:12:57.620563 29719 catalog_manager.cc:4830] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): tablet d4a5af9181d0429a8a4f4d734c59e2b6 (table rebalance_test_table_6 [id=fdf8a5db2b9b450b8db9d7a71433d20b]) successfully deleted
I20240503 02:12:57.589593 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c50aed6222af431abf62a57380647401" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:12:57.593794 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c50aed6222af431abf62a57380647401" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:12:57.594022 824 consensus_queue.cc:1035] T 5ccde7688c4e49b5b9d89d6e048c308e P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:57.629348 709 consensus_queue.cc:1035] T 00aedc4902a745d6a859cbbf99529a7f P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:57.630031 437 raft_consensus.cc:2454] T c50aed6222af431abf62a57380647401 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 0.
I20240503 02:12:57.631922 869 ts_tablet_manager.cc:1398] T c50aed6222af431abf62a57380647401 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.060s user 0.000s sys 0.002s
W20240503 02:12:57.647218 502 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.108.6:39975, user_credentials={real_user=slave}} blocked reactor thread for 47030.3us
I20240503 02:12:57.649356 876 consensus_queue.cc:1035] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:57.668705 643 consensus_queue.cc:1035] T 00aedc4902a745d6a859cbbf99529a7f P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:57.745400 29730 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:12:57.752794 307 raft_consensus.cc:1261] T c50aed6222af431abf62a57380647401 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:57.752887 437 raft_consensus.cc:3037] T c50aed6222af431abf62a57380647401 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:57.753626 876 consensus_queue.cc:1035] T c50aed6222af431abf62a57380647401 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:57.757302 878 mvcc.cc:205] Tried to move back new op lower bound from 7023420939269480448 to 7023420938691723264. Current Snapshot: MvccSnapshot[applied={T|T < 7023420939269480448}]
I20240503 02:12:57.757560 29730 catalog_manager.cc:5576] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975) does not have the latest schema for tablet c50aed6222af431abf62a57380647401 (table rebalancer_dynamic_table [id=b77a8bed065f4ea798fca0b608e4a1f1]). Expected version 1 got 0
I20240503 02:12:57.756009 437 raft_consensus.cc:1261] T c50aed6222af431abf62a57380647401 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:57.761404 821 consensus_queue.cc:1035] T c50aed6222af431abf62a57380647401 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:57.761739 899 tablet.cc:1722] T c50aed6222af431abf62a57380647401 P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:57.763374 899 tablet.cc:1620] T c50aed6222af431abf62a57380647401 P f575ce92c0224eeb80b9fae5f6561fbe: MemRowSet was empty: no flush needed.
I20240503 02:12:57.764456 29730 catalog_manager.cc:5576] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975) does not have the latest schema for tablet c50aed6222af431abf62a57380647401 (table rebalancer_dynamic_table [id=b77a8bed065f4ea798fca0b608e4a1f1]). Expected version 1 got 0
I20240503 02:12:57.764719 29727 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_extra_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:57.766791 900 tablet.cc:1722] T c50aed6222af431abf62a57380647401 P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:57.767644 900 tablet.cc:1620] T c50aed6222af431abf62a57380647401 P 8a666e798fe8477d8dd1e51b385d5ead: MemRowSet was empty: no flush needed.
I20240503 02:12:57.768906 795 tablet.cc:1722] T c50aed6222af431abf62a57380647401 P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:57.769450 795 tablet.cc:1620] T c50aed6222af431abf62a57380647401 P fbbc2be6382e47908ecb7c68c367bd26: MemRowSet was empty: no flush needed.
I20240503 02:12:57.771623 29730 catalog_manager.cc:5576] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241) does not have the latest schema for tablet c50aed6222af431abf62a57380647401 (table rebalancer_dynamic_table [id=b77a8bed065f4ea798fca0b608e4a1f1]). Expected version 1 got 0
I20240503 02:12:57.779980 417 tablet_service.cc:1467] Processing CreateTablet for tablet c2d33db64ee8482aad840af219d60b8b (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=002e01fddb8349d99a20cae11d727d7d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:57.780468 417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2d33db64ee8482aad840af219d60b8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:57.781987 546 tablet_service.cc:1467] Processing CreateTablet for tablet c2d33db64ee8482aad840af219d60b8b (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=002e01fddb8349d99a20cae11d727d7d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:57.782403 546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2d33db64ee8482aad840af219d60b8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:57.784994 29986 tablet_service.cc:1467] Processing CreateTablet for tablet 4cc7861a21f94dabb01133bac9ba0b46 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=002e01fddb8349d99a20cae11d727d7d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:57.785648 29986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cc7861a21f94dabb01133bac9ba0b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:57.786417 29853 tablet_service.cc:1467] Processing CreateTablet for tablet c2d33db64ee8482aad840af219d60b8b (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=002e01fddb8349d99a20cae11d727d7d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:57.786863 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2d33db64ee8482aad840af219d60b8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:57.789618 32621 tablet_service.cc:1467] Processing CreateTablet for tablet 4cc7861a21f94dabb01133bac9ba0b46 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=002e01fddb8349d99a20cae11d727d7d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:57.790174 32621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cc7861a21f94dabb01133bac9ba0b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:57.794121 868 tablet_bootstrap.cc:492] T c2d33db64ee8482aad840af219d60b8b P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:12:57.796413 868 tablet_bootstrap.cc:654] T c2d33db64ee8482aad840af219d60b8b P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:57.797856 869 tablet_bootstrap.cc:492] T c2d33db64ee8482aad840af219d60b8b P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:12:57.800318 869 tablet_bootstrap.cc:654] T c2d33db64ee8482aad840af219d60b8b P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:57.804127 32620 tablet_service.cc:1467] Processing CreateTablet for tablet e530c3ca50384a7c863001eb6d3fc63a (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=002e01fddb8349d99a20cae11d727d7d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:57.804605 32620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e530c3ca50384a7c863001eb6d3fc63a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:57.804777 32754 tablet_service.cc:1467] Processing CreateTablet for tablet e530c3ca50384a7c863001eb6d3fc63a (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=002e01fddb8349d99a20cae11d727d7d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:57.805227 32754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e530c3ca50384a7c863001eb6d3fc63a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:57.783239 29730 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=b77a8bed065f4ea798fca0b608e4a1f1] alter complete (version 1)
I20240503 02:12:57.813141 869 tablet_bootstrap.cc:492] T c2d33db64ee8482aad840af219d60b8b P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:12:57.813434 869 ts_tablet_manager.cc:1367] T c2d33db64ee8482aad840af219d60b8b P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.001s
I20240503 02:12:57.814442 869 raft_consensus.cc:348] T c2d33db64ee8482aad840af219d60b8b P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:57.814846 869 raft_consensus.cc:374] T c2d33db64ee8482aad840af219d60b8b P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:57.814975 869 raft_consensus.cc:729] T c2d33db64ee8482aad840af219d60b8b P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:12:57.815383 869 consensus_queue.cc:260] T c2d33db64ee8482aad840af219d60b8b P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:57.816139 869 ts_tablet_manager.cc:1398] T c2d33db64ee8482aad840af219d60b8b P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:57.822944 546 tablet_service.cc:1467] Processing CreateTablet for tablet e530c3ca50384a7c863001eb6d3fc63a (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=002e01fddb8349d99a20cae11d727d7d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:57.823516 546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e530c3ca50384a7c863001eb6d3fc63a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:57.829923 779 tablet_bootstrap.cc:492] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:12:57.832720 779 tablet_bootstrap.cc:654] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:57.835433 417 tablet_service.cc:1467] Processing CreateTablet for tablet 4cc7861a21f94dabb01133bac9ba0b46 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=002e01fddb8349d99a20cae11d727d7d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:57.836462 417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cc7861a21f94dabb01133bac9ba0b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:57.848763 911 tablet_bootstrap.cc:492] T 4cc7861a21f94dabb01133bac9ba0b46 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:57.868736 762 raft_consensus.cc:482] T c2d33db64ee8482aad840af219d60b8b P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:57.868988 762 raft_consensus.cc:504] T c2d33db64ee8482aad840af219d60b8b P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:57.869913 762 leader_election.cc:288] T c2d33db64ee8482aad840af219d60b8b P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:57.871202 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d33db64ee8482aad840af219d60b8b" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
W20240503 02:12:57.872042 370 leader_election.cc:341] T c2d33db64ee8482aad840af219d60b8b P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:57.895517 868 tablet_bootstrap.cc:492] T c2d33db64ee8482aad840af219d60b8b P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:12:57.895921 868 ts_tablet_manager.cc:1367] T c2d33db64ee8482aad840af219d60b8b P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.102s user 0.008s sys 0.004s
I20240503 02:12:57.898229 910 tablet_bootstrap.cc:492] T e530c3ca50384a7c863001eb6d3fc63a P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:12:57.902400 779 tablet_bootstrap.cc:492] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:12:57.902662 779 ts_tablet_manager.cc:1367] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.073s user 0.002s sys 0.006s
I20240503 02:12:57.903379 779 raft_consensus.cc:348] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:57.903560 779 raft_consensus.cc:374] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:57.903653 779 raft_consensus.cc:729] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:12:57.903901 779 consensus_queue.cc:260] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:57.904515 779 ts_tablet_manager.cc:1398] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:57.906697 29727 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:12:57.907959 911 tablet_bootstrap.cc:654] T 4cc7861a21f94dabb01133bac9ba0b46 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:57.910084 910 tablet_bootstrap.cc:654] T e530c3ca50384a7c863001eb6d3fc63a P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:57.914332 868 raft_consensus.cc:348] T c2d33db64ee8482aad840af219d60b8b P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:57.914743 868 raft_consensus.cc:374] T c2d33db64ee8482aad840af219d60b8b P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:57.914906 868 raft_consensus.cc:729] T c2d33db64ee8482aad840af219d60b8b P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:12:57.915205 868 consensus_queue.cc:260] T c2d33db64ee8482aad840af219d60b8b P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:57.916944 899 tablet.cc:1722] T c50aed6222af431abf62a57380647401 P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:12:57.917464 868 ts_tablet_manager.cc:1398] T c2d33db64ee8482aad840af219d60b8b P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20240503 02:12:57.918009 900 tablet.cc:1722] T c50aed6222af431abf62a57380647401 P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:12:57.918471 868 tablet_bootstrap.cc:492] T e530c3ca50384a7c863001eb6d3fc63a P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:12:57.918499 900 tablet.cc:1620] T c50aed6222af431abf62a57380647401 P 8a666e798fe8477d8dd1e51b385d5ead: MemRowSet was empty: no flush needed.
I20240503 02:12:57.918095 905 tablet_bootstrap.cc:492] T c2d33db64ee8482aad840af219d60b8b P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:57.919349 795 tablet.cc:1722] T c50aed6222af431abf62a57380647401 P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:12:57.917520 899 tablet.cc:1620] T c50aed6222af431abf62a57380647401 P f575ce92c0224eeb80b9fae5f6561fbe: MemRowSet was empty: no flush needed.
I20240503 02:12:57.921301 795 tablet.cc:1620] T c50aed6222af431abf62a57380647401 P fbbc2be6382e47908ecb7c68c367bd26: MemRowSet was empty: no flush needed.
I20240503 02:12:57.930645 868 tablet_bootstrap.cc:654] T e530c3ca50384a7c863001eb6d3fc63a P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:57.932209 905 tablet_bootstrap.cc:654] T c2d33db64ee8482aad840af219d60b8b P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:57.937953 910 tablet_bootstrap.cc:492] T e530c3ca50384a7c863001eb6d3fc63a P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:12:57.938244 910 ts_tablet_manager.cc:1367] T e530c3ca50384a7c863001eb6d3fc63a P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.000s
I20240503 02:12:57.940403 910 raft_consensus.cc:348] T e530c3ca50384a7c863001eb6d3fc63a P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:57.940749 910 raft_consensus.cc:374] T e530c3ca50384a7c863001eb6d3fc63a P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:57.940896 910 raft_consensus.cc:729] T e530c3ca50384a7c863001eb6d3fc63a P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:12:57.941205 910 consensus_queue.cc:260] T e530c3ca50384a7c863001eb6d3fc63a P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:57.941803 910 ts_tablet_manager.cc:1398] T e530c3ca50384a7c863001eb6d3fc63a P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:57.942138 910 tablet_bootstrap.cc:492] T 4cc7861a21f94dabb01133bac9ba0b46 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:12:57.943455 29734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=b77a8bed065f4ea798fca0b608e4a1f1] alter complete (version 2)
I20240503 02:12:57.944303 910 tablet_bootstrap.cc:654] T 4cc7861a21f94dabb01133bac9ba0b46 P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:57.946343 29872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d33db64ee8482aad840af219d60b8b" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
W20240503 02:12:57.947147 373 leader_election.cc:341] T c2d33db64ee8482aad840af219d60b8b P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:57.947360 373 leader_election.cc:302] T c2d33db64ee8482aad840af219d60b8b P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 16440e365430405fb10366ab70e2b429, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:12:57.947696 762 raft_consensus.cc:2726] T c2d33db64ee8482aad840af219d60b8b P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:57.949147 911 tablet_bootstrap.cc:492] T 4cc7861a21f94dabb01133bac9ba0b46 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:57.949373 911 ts_tablet_manager.cc:1367] T 4cc7861a21f94dabb01133bac9ba0b46 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.101s user 0.004s sys 0.000s
I20240503 02:12:57.949581 905 tablet_bootstrap.cc:492] T c2d33db64ee8482aad840af219d60b8b P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:57.949735 905 ts_tablet_manager.cc:1367] T c2d33db64ee8482aad840af219d60b8b P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
I20240503 02:12:57.949982 869 tablet_bootstrap.cc:492] T 4cc7861a21f94dabb01133bac9ba0b46 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:12:57.951589 869 tablet_bootstrap.cc:654] T 4cc7861a21f94dabb01133bac9ba0b46 P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:57.952452 911 raft_consensus.cc:348] T 4cc7861a21f94dabb01133bac9ba0b46 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:57.953325 911 raft_consensus.cc:374] T 4cc7861a21f94dabb01133bac9ba0b46 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:57.953490 911 raft_consensus.cc:729] T 4cc7861a21f94dabb01133bac9ba0b46 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:57.953805 911 consensus_queue.cc:260] T 4cc7861a21f94dabb01133bac9ba0b46 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:57.957726 911 ts_tablet_manager.cc:1398] T 4cc7861a21f94dabb01133bac9ba0b46 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:12:57.966435 868 tablet_bootstrap.cc:492] T e530c3ca50384a7c863001eb6d3fc63a P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:12:57.966642 868 ts_tablet_manager.cc:1367] T e530c3ca50384a7c863001eb6d3fc63a P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.048s user 0.002s sys 0.001s
I20240503 02:12:57.967621 868 raft_consensus.cc:348] T e530c3ca50384a7c863001eb6d3fc63a P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:57.967906 868 raft_consensus.cc:374] T e530c3ca50384a7c863001eb6d3fc63a P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:57.968014 868 raft_consensus.cc:729] T e530c3ca50384a7c863001eb6d3fc63a P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:12:57.968281 868 consensus_queue.cc:260] T e530c3ca50384a7c863001eb6d3fc63a P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:57.970402 905 raft_consensus.cc:348] T c2d33db64ee8482aad840af219d60b8b P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:57.970573 868 ts_tablet_manager.cc:1398] T e530c3ca50384a7c863001eb6d3fc63a P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240503 02:12:57.971397 905 raft_consensus.cc:374] T c2d33db64ee8482aad840af219d60b8b P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:57.975623 905 raft_consensus.cc:729] T c2d33db64ee8482aad840af219d60b8b P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:57.975983 905 consensus_queue.cc:260] T c2d33db64ee8482aad840af219d60b8b P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:57.976691 905 ts_tablet_manager.cc:1398] T c2d33db64ee8482aad840af219d60b8b P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:12:57.981844 910 tablet_bootstrap.cc:492] T 4cc7861a21f94dabb01133bac9ba0b46 P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:12:57.983671 869 tablet_bootstrap.cc:492] T 4cc7861a21f94dabb01133bac9ba0b46 P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:12:57.984164 869 ts_tablet_manager.cc:1367] T 4cc7861a21f94dabb01133bac9ba0b46 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.010s
I20240503 02:12:57.985256 869 raft_consensus.cc:348] T 4cc7861a21f94dabb01133bac9ba0b46 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:57.985554 869 raft_consensus.cc:374] T 4cc7861a21f94dabb01133bac9ba0b46 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:57.985687 869 raft_consensus.cc:729] T 4cc7861a21f94dabb01133bac9ba0b46 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:12:57.985978 869 consensus_queue.cc:260] T 4cc7861a21f94dabb01133bac9ba0b46 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:57.987569 869 ts_tablet_manager.cc:1398] T 4cc7861a21f94dabb01133bac9ba0b46 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:12:57.988922 910 ts_tablet_manager.cc:1367] T 4cc7861a21f94dabb01133bac9ba0b46 P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.040s user 0.010s sys 0.000s
I20240503 02:12:57.990015 29734 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240503 02:12:57.990259 29734 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240503 02:12:57.992450 910 raft_consensus.cc:348] T 4cc7861a21f94dabb01133bac9ba0b46 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:57.992707 910 raft_consensus.cc:374] T 4cc7861a21f94dabb01133bac9ba0b46 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:57.992777 910 raft_consensus.cc:729] T 4cc7861a21f94dabb01133bac9ba0b46 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:12:57.994516 910 consensus_queue.cc:260] T 4cc7861a21f94dabb01133bac9ba0b46 P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:57.995075 910 ts_tablet_manager.cc:1398] T 4cc7861a21f94dabb01133bac9ba0b46 P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 02:12:57.996654 29734 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet d983c9264e5f475aa6d71d3f7eff758f
I20240503 02:12:57.997598 29853 tablet_service.cc:1514] Processing DeleteTablet for tablet d983c9264e5f475aa6d71d3f7eff758f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:57 UTC) from {username='slave'} at 127.0.0.1:58744
I20240503 02:12:57.997690 29986 tablet_service.cc:1514] Processing DeleteTablet for tablet d983c9264e5f475aa6d71d3f7eff758f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:57 UTC) from {username='slave'} at 127.0.0.1:52158
I20240503 02:12:57.997958 659 tablet_replica.cc:330] T d983c9264e5f475aa6d71d3f7eff758f P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:12:57.998035 657 tablet_replica.cc:330] T d983c9264e5f475aa6d71d3f7eff758f P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:12:57.998168 659 raft_consensus.cc:2229] T d983c9264e5f475aa6d71d3f7eff758f P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:57.998345 657 raft_consensus.cc:2229] T d983c9264e5f475aa6d71d3f7eff758f P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:57.998378 659 raft_consensus.cc:2258] T d983c9264e5f475aa6d71d3f7eff758f P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:57.998689 657 raft_consensus.cc:2258] T d983c9264e5f475aa6d71d3f7eff758f P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:57.999107 659 ts_tablet_manager.cc:1875] T d983c9264e5f475aa6d71d3f7eff758f P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:57.999691 657 ts_tablet_manager.cc:1875] T d983c9264e5f475aa6d71d3f7eff758f P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:58.008108 659 ts_tablet_manager.cc:1888] T d983c9264e5f475aa6d71d3f7eff758f P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:58.008265 659 log.cc:1196] T d983c9264e5f475aa6d71d3f7eff758f P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/d983c9264e5f475aa6d71d3f7eff758f
I20240503 02:12:58.008652 659 ts_tablet_manager.cc:1909] T d983c9264e5f475aa6d71d3f7eff758f P 16440e365430405fb10366ab70e2b429: Deleting consensus metadata
I20240503 02:12:58.009402 643 raft_consensus.cc:482] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:58.009682 29734 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet b8a9660c16e34a2098af2ce527810583
I20240503 02:12:58.009622 643 raft_consensus.cc:504] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:58.009918 30114 tablet_service.cc:1514] Processing DeleteTablet for tablet d983c9264e5f475aa6d71d3f7eff758f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:57 UTC) from {username='slave'} at 127.0.0.1:44812
I20240503 02:12:58.010195 796 tablet_replica.cc:330] T d983c9264e5f475aa6d71d3f7eff758f P a9094240860e43bfb129193f3bfd5be6: stopping tablet replica
I20240503 02:12:58.010207 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet d983c9264e5f475aa6d71d3f7eff758f (table rebalance_test_table_8 [id=67dd4000d7e143e4b042a4aa61eac02c]) successfully deleted
I20240503 02:12:58.010358 643 leader_election.cc:288] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:12:58.010413 796 raft_consensus.cc:2229] T d983c9264e5f475aa6d71d3f7eff758f P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:58.010504 30244 tablet_service.cc:1514] Processing DeleteTablet for tablet b8a9660c16e34a2098af2ce527810583 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:57 UTC) from {username='slave'} at 127.0.0.1:49856
I20240503 02:12:58.010776 656 tablet_replica.cc:330] T b8a9660c16e34a2098af2ce527810583 P 1f10f80f13c84e80b03eda7085a3b012: stopping tablet replica
I20240503 02:12:58.010777 796 raft_consensus.cc:2258] T d983c9264e5f475aa6d71d3f7eff758f P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:58.010970 656 raft_consensus.cc:2229] T b8a9660c16e34a2098af2ce527810583 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:58.011294 656 raft_consensus.cc:2258] T b8a9660c16e34a2098af2ce527810583 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:58.011247 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e530c3ca50384a7c863001eb6d3fc63a" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:12:58.011503 796 ts_tablet_manager.cc:1875] T d983c9264e5f475aa6d71d3f7eff758f P a9094240860e43bfb129193f3bfd5be6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:58.011482 32641 raft_consensus.cc:2454] T e530c3ca50384a7c863001eb6d3fc63a P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:12:58.011674 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e530c3ca50384a7c863001eb6d3fc63a" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:12:58.011898 567 raft_consensus.cc:2454] T e530c3ca50384a7c863001eb6d3fc63a P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:12:58.012001 32709 leader_election.cc:302] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, 9a4d92bc25374fb3960fde7d10be309d; no voters:
I20240503 02:12:58.012061 656 ts_tablet_manager.cc:1875] T b8a9660c16e34a2098af2ce527810583 P 1f10f80f13c84e80b03eda7085a3b012: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:58.012346 29986 tablet_service.cc:1514] Processing DeleteTablet for tablet b8a9660c16e34a2098af2ce527810583 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:57 UTC) from {username='slave'} at 127.0.0.1:52158
I20240503 02:12:58.012595 643 raft_consensus.cc:2781] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:58.012686 643 raft_consensus.cc:482] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:58.012792 643 raft_consensus.cc:3037] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:58.015417 643 raft_consensus.cc:504] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:58.016561 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e530c3ca50384a7c863001eb6d3fc63a" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:12:58.016782 567 raft_consensus.cc:3037] T e530c3ca50384a7c863001eb6d3fc63a P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:58.017788 796 ts_tablet_manager.cc:1888] T d983c9264e5f475aa6d71d3f7eff758f P a9094240860e43bfb129193f3bfd5be6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:58.017925 796 log.cc:1196] T d983c9264e5f475aa6d71d3f7eff758f P a9094240860e43bfb129193f3bfd5be6: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/wals/d983c9264e5f475aa6d71d3f7eff758f
I20240503 02:12:58.017918 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e530c3ca50384a7c863001eb6d3fc63a" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d"
I20240503 02:12:58.018141 32641 raft_consensus.cc:3037] T e530c3ca50384a7c863001eb6d3fc63a P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:58.018383 796 ts_tablet_manager.cc:1909] T d983c9264e5f475aa6d71d3f7eff758f P a9094240860e43bfb129193f3bfd5be6: Deleting consensus metadata
I20240503 02:12:58.018673 656 ts_tablet_manager.cc:1888] T b8a9660c16e34a2098af2ce527810583 P 1f10f80f13c84e80b03eda7085a3b012: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:58.018839 656 log.cc:1196] T b8a9660c16e34a2098af2ce527810583 P 1f10f80f13c84e80b03eda7085a3b012: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/wals/b8a9660c16e34a2098af2ce527810583
I20240503 02:12:58.019482 29718 catalog_manager.cc:4830] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): tablet d983c9264e5f475aa6d71d3f7eff758f (table rebalance_test_table_8 [id=67dd4000d7e143e4b042a4aa61eac02c]) successfully deleted
I20240503 02:12:58.019733 656 ts_tablet_manager.cc:1909] T b8a9660c16e34a2098af2ce527810583 P 1f10f80f13c84e80b03eda7085a3b012: Deleting consensus metadata
I20240503 02:12:58.019754 567 raft_consensus.cc:2454] T e530c3ca50384a7c863001eb6d3fc63a P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:12:58.020984 29719 catalog_manager.cc:4830] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): tablet b8a9660c16e34a2098af2ce527810583 (table rebalance_test_table_8 [id=67dd4000d7e143e4b042a4aa61eac02c]) successfully deleted
I20240503 02:12:58.021212 32708 leader_election.cc:302] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:12:58.021500 709 raft_consensus.cc:2781] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:58.023372 709 raft_consensus.cc:686] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Becoming Leader. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Running, Role: LEADER
I20240503 02:12:58.023453 643 leader_election.cc:288] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Requested vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:12:58.023690 709 consensus_queue.cc:237] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:58.024560 32641 raft_consensus.cc:2454] T e530c3ca50384a7c863001eb6d3fc63a P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:12:58.025887 29734 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet f511ffcfb1d448a89913dbe432bf83ce
I20240503 02:12:58.026494 29853 tablet_service.cc:1514] Processing DeleteTablet for tablet b8a9660c16e34a2098af2ce527810583 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:57 UTC) from {username='slave'} at 127.0.0.1:58744
I20240503 02:12:58.026880 30244 tablet_service.cc:1514] Processing DeleteTablet for tablet f511ffcfb1d448a89913dbe432bf83ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:57 UTC) from {username='slave'} at 127.0.0.1:49856
I20240503 02:12:58.027544 659 tablet_replica.cc:330] T b8a9660c16e34a2098af2ce527810583 P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:12:58.027539 29986 tablet_service.cc:1514] Processing DeleteTablet for tablet f511ffcfb1d448a89913dbe432bf83ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:57 UTC) from {username='slave'} at 127.0.0.1:52158
I20240503 02:12:58.028388 659 raft_consensus.cc:2229] T b8a9660c16e34a2098af2ce527810583 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:58.028734 659 raft_consensus.cc:2258] T b8a9660c16e34a2098af2ce527810583 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:58.029618 659 ts_tablet_manager.cc:1875] T b8a9660c16e34a2098af2ce527810583 P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:58.029762 30114 tablet_service.cc:1514] Processing DeleteTablet for tablet f511ffcfb1d448a89913dbe432bf83ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:57 UTC) from {username='slave'} at 127.0.0.1:44812
I20240503 02:12:58.030018 796 tablet_replica.cc:330] T f511ffcfb1d448a89913dbe432bf83ce P a9094240860e43bfb129193f3bfd5be6: stopping tablet replica
I20240503 02:12:58.030230 796 raft_consensus.cc:2229] T f511ffcfb1d448a89913dbe432bf83ce P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:58.030443 796 raft_consensus.cc:2258] T f511ffcfb1d448a89913dbe432bf83ce P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:58.031170 796 ts_tablet_manager.cc:1875] T f511ffcfb1d448a89913dbe432bf83ce P a9094240860e43bfb129193f3bfd5be6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:58.035251 762 raft_consensus.cc:482] T 4cc7861a21f94dabb01133bac9ba0b46 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:58.035429 762 raft_consensus.cc:504] T 4cc7861a21f94dabb01133bac9ba0b46 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:58.036165 762 leader_election.cc:288] T 4cc7861a21f94dabb01133bac9ba0b46 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:12:58.037197 659 ts_tablet_manager.cc:1888] T b8a9660c16e34a2098af2ce527810583 P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:58.037348 659 log.cc:1196] T b8a9660c16e34a2098af2ce527810583 P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/b8a9660c16e34a2098af2ce527810583
I20240503 02:12:58.037374 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc7861a21f94dabb01133bac9ba0b46" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:12:58.037765 659 ts_tablet_manager.cc:1909] T b8a9660c16e34a2098af2ce527810583 P 16440e365430405fb10366ab70e2b429: Deleting consensus metadata
I20240503 02:12:58.037726 32641 raft_consensus.cc:2454] T 4cc7861a21f94dabb01133bac9ba0b46 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 0.
I20240503 02:12:58.038156 657 ts_tablet_manager.cc:1888] T d983c9264e5f475aa6d71d3f7eff758f P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:58.038210 371 leader_election.cc:302] T 4cc7861a21f94dabb01133bac9ba0b46 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:12:58.038321 657 log.cc:1196] T d983c9264e5f475aa6d71d3f7eff758f P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/d983c9264e5f475aa6d71d3f7eff758f
I20240503 02:12:58.038579 762 raft_consensus.cc:2781] T 4cc7861a21f94dabb01133bac9ba0b46 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:58.038676 762 raft_consensus.cc:482] T 4cc7861a21f94dabb01133bac9ba0b46 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:58.038791 762 raft_consensus.cc:3037] T 4cc7861a21f94dabb01133bac9ba0b46 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:58.038929 657 ts_tablet_manager.cc:1909] T d983c9264e5f475aa6d71d3f7eff758f P 155735c7f8b14997ac4c770aee2a9401: Deleting consensus metadata
I20240503 02:12:58.038952 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet b8a9660c16e34a2098af2ce527810583 (table rebalance_test_table_8 [id=67dd4000d7e143e4b042a4aa61eac02c]) successfully deleted
I20240503 02:12:58.041169 796 ts_tablet_manager.cc:1888] T f511ffcfb1d448a89913dbe432bf83ce P a9094240860e43bfb129193f3bfd5be6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:58.041323 796 log.cc:1196] T f511ffcfb1d448a89913dbe432bf83ce P a9094240860e43bfb129193f3bfd5be6: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/wals/f511ffcfb1d448a89913dbe432bf83ce
I20240503 02:12:58.041416 762 raft_consensus.cc:504] T 4cc7861a21f94dabb01133bac9ba0b46 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:58.041659 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet d983c9264e5f475aa6d71d3f7eff758f (table rebalance_test_table_8 [id=67dd4000d7e143e4b042a4aa61eac02c]) successfully deleted
I20240503 02:12:58.041744 796 ts_tablet_manager.cc:1909] T f511ffcfb1d448a89913dbe432bf83ce P a9094240860e43bfb129193f3bfd5be6: Deleting consensus metadata
I20240503 02:12:58.042064 657 tablet_replica.cc:330] T b8a9660c16e34a2098af2ce527810583 P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:12:58.042342 657 raft_consensus.cc:2229] T b8a9660c16e34a2098af2ce527810583 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:58.042394 656 tablet_replica.cc:330] T f511ffcfb1d448a89913dbe432bf83ce P 1f10f80f13c84e80b03eda7085a3b012: stopping tablet replica
I20240503 02:12:58.042641 656 raft_consensus.cc:2229] T f511ffcfb1d448a89913dbe432bf83ce P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:58.043035 656 raft_consensus.cc:2258] T f511ffcfb1d448a89913dbe432bf83ce P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:58.043511 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc7861a21f94dabb01133bac9ba0b46" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d"
I20240503 02:12:58.043787 32641 raft_consensus.cc:3037] T 4cc7861a21f94dabb01133bac9ba0b46 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:58.043960 656 ts_tablet_manager.cc:1875] T f511ffcfb1d448a89913dbe432bf83ce P 1f10f80f13c84e80b03eda7085a3b012: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:58.044080 657 raft_consensus.cc:2258] T b8a9660c16e34a2098af2ce527810583 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:58.043336 29734 catalog_manager.cc:5484] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6). New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:12:58.046895 762 leader_election.cc:288] T 4cc7861a21f94dabb01133bac9ba0b46 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Requested vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:12:58.047403 29727 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:12:58.048134 32641 raft_consensus.cc:2454] T 4cc7861a21f94dabb01133bac9ba0b46 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 1.
I20240503 02:12:58.048619 371 leader_election.cc:302] T 4cc7861a21f94dabb01133bac9ba0b46 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:12:58.048965 824 raft_consensus.cc:2781] T 4cc7861a21f94dabb01133bac9ba0b46 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:58.049167 824 raft_consensus.cc:686] T 4cc7861a21f94dabb01133bac9ba0b46 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Becoming Leader. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Running, Role: LEADER
I20240503 02:12:58.049433 824 consensus_queue.cc:237] T 4cc7861a21f94dabb01133bac9ba0b46 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:58.050873 656 ts_tablet_manager.cc:1888] T f511ffcfb1d448a89913dbe432bf83ce P 1f10f80f13c84e80b03eda7085a3b012: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:58.051075 656 log.cc:1196] T f511ffcfb1d448a89913dbe432bf83ce P 1f10f80f13c84e80b03eda7085a3b012: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/wals/f511ffcfb1d448a89913dbe432bf83ce
I20240503 02:12:58.051265 657 ts_tablet_manager.cc:1875] T b8a9660c16e34a2098af2ce527810583 P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:58.051734 656 ts_tablet_manager.cc:1909] T f511ffcfb1d448a89913dbe432bf83ce P 1f10f80f13c84e80b03eda7085a3b012: Deleting consensus metadata
I20240503 02:12:58.052981 29719 catalog_manager.cc:4830] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): tablet f511ffcfb1d448a89913dbe432bf83ce (table rebalance_test_table_8 [id=67dd4000d7e143e4b042a4aa61eac02c]) successfully deleted
I20240503 02:12:58.054991 29718 catalog_manager.cc:4830] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): tablet f511ffcfb1d448a89913dbe432bf83ce (table rebalance_test_table_8 [id=67dd4000d7e143e4b042a4aa61eac02c]) successfully deleted
I20240503 02:12:58.057318 899 tablet.cc:1722] T c50aed6222af431abf62a57380647401 P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:12:58.061226 795 tablet.cc:1722] T c50aed6222af431abf62a57380647401 P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:12:58.062016 900 tablet.cc:1722] T c50aed6222af431abf62a57380647401 P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:12:58.073546 29734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table_new_name [id=b77a8bed065f4ea798fca0b608e4a1f1] alter complete (version 3)
I20240503 02:12:58.079044 29734 catalog_manager.cc:5484] T 4cc7861a21f94dabb01133bac9ba0b46 P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7). New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:12:58.087328 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc7861a21f94dabb01133bac9ba0b46" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:12:58.087603 30004 raft_consensus.cc:2454] T 4cc7861a21f94dabb01133bac9ba0b46 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 0.
I20240503 02:12:58.087510 30006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc7861a21f94dabb01133bac9ba0b46" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:12:58.087785 30006 raft_consensus.cc:3037] T 4cc7861a21f94dabb01133bac9ba0b46 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:58.091843 30006 raft_consensus.cc:2454] T 4cc7861a21f94dabb01133bac9ba0b46 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 1.
I20240503 02:12:58.094729 657 ts_tablet_manager.cc:1888] T b8a9660c16e34a2098af2ce527810583 P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:58.094866 657 log.cc:1196] T b8a9660c16e34a2098af2ce527810583 P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/b8a9660c16e34a2098af2ce527810583
I20240503 02:12:58.095237 657 ts_tablet_manager.cc:1909] T b8a9660c16e34a2098af2ce527810583 P 155735c7f8b14997ac4c770aee2a9401: Deleting consensus metadata
I20240503 02:12:58.096207 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet b8a9660c16e34a2098af2ce527810583 (table rebalance_test_table_8 [id=67dd4000d7e143e4b042a4aa61eac02c]) successfully deleted
I20240503 02:12:58.099987 657 tablet_replica.cc:330] T f511ffcfb1d448a89913dbe432bf83ce P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:12:58.103266 657 raft_consensus.cc:2229] T f511ffcfb1d448a89913dbe432bf83ce P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:58.103565 657 raft_consensus.cc:2258] T f511ffcfb1d448a89913dbe432bf83ce P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:58.105289 657 ts_tablet_manager.cc:1875] T f511ffcfb1d448a89913dbe432bf83ce P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:58.113541 657 ts_tablet_manager.cc:1888] T f511ffcfb1d448a89913dbe432bf83ce P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:58.113765 657 log.cc:1196] T f511ffcfb1d448a89913dbe432bf83ce P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/f511ffcfb1d448a89913dbe432bf83ce
I20240503 02:12:58.114192 657 ts_tablet_manager.cc:1909] T f511ffcfb1d448a89913dbe432bf83ce P 155735c7f8b14997ac4c770aee2a9401: Deleting consensus metadata
I20240503 02:12:58.115298 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet f511ffcfb1d448a89913dbe432bf83ce (table rebalance_test_table_8 [id=67dd4000d7e143e4b042a4aa61eac02c]) successfully deleted
I20240503 02:12:58.174263 29729 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:12:58.174485 29729 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:12:58.178200 29729 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet c50aed6222af431abf62a57380647401
I20240503 02:12:58.179699 32754 tablet_service.cc:1514] Processing DeleteTablet for tablet c50aed6222af431abf62a57380647401 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:58 UTC) from {username='slave'} at 127.0.0.1:46360
I20240503 02:12:58.179807 546 tablet_service.cc:1514] Processing DeleteTablet for tablet c50aed6222af431abf62a57380647401 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:58 UTC) from {username='slave'} at 127.0.0.1:39028
I20240503 02:12:58.180408 417 tablet_service.cc:1514] Processing DeleteTablet for tablet c50aed6222af431abf62a57380647401 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:58 UTC) from {username='slave'} at 127.0.0.1:50814
I20240503 02:12:58.197075 951 tablet_replica.cc:330] T c50aed6222af431abf62a57380647401 P 8a666e798fe8477d8dd1e51b385d5ead: stopping tablet replica
I20240503 02:12:58.197540 951 raft_consensus.cc:2229] T c50aed6222af431abf62a57380647401 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:58.197880 951 raft_consensus.cc:2258] T c50aed6222af431abf62a57380647401 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:58.199106 951 ts_tablet_manager.cc:1875] T c50aed6222af431abf62a57380647401 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:58.213748 951 ts_tablet_manager.cc:1888] T c50aed6222af431abf62a57380647401 P 8a666e798fe8477d8dd1e51b385d5ead: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:58.214071 951 log.cc:1196] T c50aed6222af431abf62a57380647401 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal/wals/c50aed6222af431abf62a57380647401
I20240503 02:12:58.215332 951 ts_tablet_manager.cc:1909] T c50aed6222af431abf62a57380647401 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting consensus metadata
I20240503 02:12:58.217150 29719 catalog_manager.cc:4830] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): tablet c50aed6222af431abf62a57380647401 (table rebalancer_dynamic_table_new_name [id=b77a8bed065f4ea798fca0b608e4a1f1]) successfully deleted
I20240503 02:12:58.222411 952 tablet_replica.cc:330] T c50aed6222af431abf62a57380647401 P f575ce92c0224eeb80b9fae5f6561fbe: stopping tablet replica
I20240503 02:12:58.222990 953 tablet_replica.cc:330] T c50aed6222af431abf62a57380647401 P fbbc2be6382e47908ecb7c68c367bd26: stopping tablet replica
I20240503 02:12:58.224375 952 raft_consensus.cc:2229] T c50aed6222af431abf62a57380647401 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:58.224649 953 raft_consensus.cc:2229] T c50aed6222af431abf62a57380647401 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:58.224947 953 raft_consensus.cc:2258] T c50aed6222af431abf62a57380647401 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:58.225816 952 raft_consensus.cc:2258] T c50aed6222af431abf62a57380647401 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:58.227144 953 ts_tablet_manager.cc:1875] T c50aed6222af431abf62a57380647401 P fbbc2be6382e47908ecb7c68c367bd26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:58.228540 952 ts_tablet_manager.cc:1875] T c50aed6222af431abf62a57380647401 P f575ce92c0224eeb80b9fae5f6561fbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:58.234793 952 ts_tablet_manager.cc:1888] T c50aed6222af431abf62a57380647401 P f575ce92c0224eeb80b9fae5f6561fbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:58.234966 952 log.cc:1196] T c50aed6222af431abf62a57380647401 P f575ce92c0224eeb80b9fae5f6561fbe: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal/wals/c50aed6222af431abf62a57380647401
I20240503 02:12:58.235388 952 ts_tablet_manager.cc:1909] T c50aed6222af431abf62a57380647401 P f575ce92c0224eeb80b9fae5f6561fbe: Deleting consensus metadata
I20240503 02:12:58.236739 29718 catalog_manager.cc:4830] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): tablet c50aed6222af431abf62a57380647401 (table rebalancer_dynamic_table_new_name [id=b77a8bed065f4ea798fca0b608e4a1f1]) successfully deleted
I20240503 02:12:58.241603 953 ts_tablet_manager.cc:1888] T c50aed6222af431abf62a57380647401 P fbbc2be6382e47908ecb7c68c367bd26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:58.241731 953 log.cc:1196] T c50aed6222af431abf62a57380647401 P fbbc2be6382e47908ecb7c68c367bd26: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal/wals/c50aed6222af431abf62a57380647401
I20240503 02:12:58.242118 953 ts_tablet_manager.cc:1909] T c50aed6222af431abf62a57380647401 P fbbc2be6382e47908ecb7c68c367bd26: Deleting consensus metadata
I20240503 02:12:58.242966 29719 catalog_manager.cc:4830] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): tablet c50aed6222af431abf62a57380647401 (table rebalancer_dynamic_table_new_name [id=b77a8bed065f4ea798fca0b608e4a1f1]) successfully deleted
I20240503 02:12:58.287376 29731 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:12:58.294610 32754 tablet_service.cc:1467] Processing CreateTablet for tablet def0775bc6eb4ae8b185ef113c276955 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=59b3767ebde244f981465e9656416610]), partition=
I20240503 02:12:58.294597 32620 tablet_service.cc:1467] Processing CreateTablet for tablet def0775bc6eb4ae8b185ef113c276955 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=59b3767ebde244f981465e9656416610]), partition=
I20240503 02:12:58.294708 546 tablet_service.cc:1467] Processing CreateTablet for tablet def0775bc6eb4ae8b185ef113c276955 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=59b3767ebde244f981465e9656416610]), partition=
I20240503 02:12:58.295001 32754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet def0775bc6eb4ae8b185ef113c276955. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:58.295065 546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet def0775bc6eb4ae8b185ef113c276955. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:58.295089 32620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet def0775bc6eb4ae8b185ef113c276955. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:58.301375 868 tablet_bootstrap.cc:492] T def0775bc6eb4ae8b185ef113c276955 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:12:58.303027 868 tablet_bootstrap.cc:654] T def0775bc6eb4ae8b185ef113c276955 P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:58.303774 779 tablet_bootstrap.cc:492] T def0775bc6eb4ae8b185ef113c276955 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:12:58.304265 910 tablet_bootstrap.cc:492] T def0775bc6eb4ae8b185ef113c276955 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:12:58.305653 779 tablet_bootstrap.cc:654] T def0775bc6eb4ae8b185ef113c276955 P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:58.305816 910 tablet_bootstrap.cc:654] T def0775bc6eb4ae8b185ef113c276955 P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:58.306116 868 tablet_bootstrap.cc:492] T def0775bc6eb4ae8b185ef113c276955 P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:12:58.306311 868 ts_tablet_manager.cc:1367] T def0775bc6eb4ae8b185ef113c276955 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240503 02:12:58.307157 868 raft_consensus.cc:348] T def0775bc6eb4ae8b185ef113c276955 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:58.307562 868 raft_consensus.cc:374] T def0775bc6eb4ae8b185ef113c276955 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:58.307698 868 raft_consensus.cc:729] T def0775bc6eb4ae8b185ef113c276955 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:12:58.308051 868 consensus_queue.cc:260] T def0775bc6eb4ae8b185ef113c276955 P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:58.308760 868 ts_tablet_manager.cc:1398] T def0775bc6eb4ae8b185ef113c276955 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:12:58.308840 779 tablet_bootstrap.cc:492] T def0775bc6eb4ae8b185ef113c276955 P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:12:58.309018 779 ts_tablet_manager.cc:1367] T def0775bc6eb4ae8b185ef113c276955 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:12:58.309650 910 tablet_bootstrap.cc:492] T def0775bc6eb4ae8b185ef113c276955 P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:12:58.309800 910 ts_tablet_manager.cc:1367] T def0775bc6eb4ae8b185ef113c276955 P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 02:12:58.309808 779 raft_consensus.cc:348] T def0775bc6eb4ae8b185ef113c276955 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:58.310038 779 raft_consensus.cc:374] T def0775bc6eb4ae8b185ef113c276955 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:58.310117 779 raft_consensus.cc:729] T def0775bc6eb4ae8b185ef113c276955 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:12:58.310388 779 consensus_queue.cc:260] T def0775bc6eb4ae8b185ef113c276955 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:58.310837 910 raft_consensus.cc:348] T def0775bc6eb4ae8b185ef113c276955 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:58.311045 910 raft_consensus.cc:374] T def0775bc6eb4ae8b185ef113c276955 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:58.311151 910 raft_consensus.cc:729] T def0775bc6eb4ae8b185ef113c276955 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:12:58.311389 910 consensus_queue.cc:260] T def0775bc6eb4ae8b185ef113c276955 P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:58.312206 910 ts_tablet_manager.cc:1398] T def0775bc6eb4ae8b185ef113c276955 P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:58.313627 779 ts_tablet_manager.cc:1398] T def0775bc6eb4ae8b185ef113c276955 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:12:58.319454 709 raft_consensus.cc:482] T def0775bc6eb4ae8b185ef113c276955 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:58.319633 709 raft_consensus.cc:504] T def0775bc6eb4ae8b185ef113c276955 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:58.320204 709 leader_election.cc:288] T def0775bc6eb4ae8b185ef113c276955 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:12:58.320645 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "def0775bc6eb4ae8b185ef113c276955" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:12:58.320932 567 raft_consensus.cc:2454] T def0775bc6eb4ae8b185ef113c276955 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:12:58.320880 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "def0775bc6eb4ae8b185ef113c276955" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:12:58.321349 32708 leader_election.cc:302] T def0775bc6eb4ae8b185ef113c276955 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:12:58.321722 32641 raft_consensus.cc:2454] T def0775bc6eb4ae8b185ef113c276955 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:12:58.322018 709 raft_consensus.cc:2781] T def0775bc6eb4ae8b185ef113c276955 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:58.322139 709 raft_consensus.cc:482] T def0775bc6eb4ae8b185ef113c276955 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:58.322256 709 raft_consensus.cc:3037] T def0775bc6eb4ae8b185ef113c276955 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:58.324656 709 raft_consensus.cc:504] T def0775bc6eb4ae8b185ef113c276955 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:58.325325 709 leader_election.cc:288] T def0775bc6eb4ae8b185ef113c276955 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Requested vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:12:58.325860 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "def0775bc6eb4ae8b185ef113c276955" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d"
I20240503 02:12:58.325984 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "def0775bc6eb4ae8b185ef113c276955" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:12:58.326093 32641 raft_consensus.cc:3037] T def0775bc6eb4ae8b185ef113c276955 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:58.326215 567 raft_consensus.cc:3037] T def0775bc6eb4ae8b185ef113c276955 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:58.328049 32641 raft_consensus.cc:2454] T def0775bc6eb4ae8b185ef113c276955 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:12:58.328347 567 raft_consensus.cc:2454] T def0775bc6eb4ae8b185ef113c276955 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:12:58.328388 32709 leader_election.cc:302] T def0775bc6eb4ae8b185ef113c276955 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, 9a4d92bc25374fb3960fde7d10be309d; no voters:
I20240503 02:12:58.328675 709 raft_consensus.cc:2781] T def0775bc6eb4ae8b185ef113c276955 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:58.328984 709 raft_consensus.cc:686] T def0775bc6eb4ae8b185ef113c276955 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Becoming Leader. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Running, Role: LEADER
I20240503 02:12:58.329310 709 consensus_queue.cc:237] T def0775bc6eb4ae8b185ef113c276955 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:58.331960 29725 catalog_manager.cc:5484] T def0775bc6eb4ae8b185ef113c276955 P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6). New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:12:58.413595 877 raft_consensus.cc:482] T c2d33db64ee8482aad840af219d60b8b P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:58.413801 877 raft_consensus.cc:504] T c2d33db64ee8482aad840af219d60b8b P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:58.414721 877 leader_election.cc:288] T c2d33db64ee8482aad840af219d60b8b P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:58.415228 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d33db64ee8482aad840af219d60b8b" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:12:58.415524 437 raft_consensus.cc:2454] T c2d33db64ee8482aad840af219d60b8b P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 0.
I20240503 02:12:58.416081 502 leader_election.cc:302] T c2d33db64ee8482aad840af219d60b8b P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f575ce92c0224eeb80b9fae5f6561fbe, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:12:58.416358 877 raft_consensus.cc:2781] T c2d33db64ee8482aad840af219d60b8b P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:58.416599 877 raft_consensus.cc:482] T c2d33db64ee8482aad840af219d60b8b P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:58.416733 877 raft_consensus.cc:3037] T c2d33db64ee8482aad840af219d60b8b P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:58.419718 877 raft_consensus.cc:504] T c2d33db64ee8482aad840af219d60b8b P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:58.420806 877 leader_election.cc:288] T c2d33db64ee8482aad840af219d60b8b P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 election: Requested vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:12:58.421108 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d33db64ee8482aad840af219d60b8b" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:12:58.421375 437 raft_consensus.cc:3037] T c2d33db64ee8482aad840af219d60b8b P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:58.422236 29873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d33db64ee8482aad840af219d60b8b" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:12:58.422526 29873 raft_consensus.cc:2454] T c2d33db64ee8482aad840af219d60b8b P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 0.
I20240503 02:12:58.422490 29872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d33db64ee8482aad840af219d60b8b" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:12:58.422751 29872 raft_consensus.cc:3037] T c2d33db64ee8482aad840af219d60b8b P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:58.424238 437 raft_consensus.cc:2454] T c2d33db64ee8482aad840af219d60b8b P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 1.
I20240503 02:12:58.425637 29872 raft_consensus.cc:2454] T c2d33db64ee8482aad840af219d60b8b P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 1.
I20240503 02:12:58.426792 502 leader_election.cc:302] T c2d33db64ee8482aad840af219d60b8b P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f575ce92c0224eeb80b9fae5f6561fbe, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:12:58.427139 877 raft_consensus.cc:2781] T c2d33db64ee8482aad840af219d60b8b P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:58.427490 877 raft_consensus.cc:686] T c2d33db64ee8482aad840af219d60b8b P f575ce92c0224eeb80b9fae5f6561fbe [term 1 LEADER]: Becoming Leader. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Running, Role: LEADER
I20240503 02:12:58.427839 877 consensus_queue.cc:237] T c2d33db64ee8482aad840af219d60b8b P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:58.431869 29731 catalog_manager.cc:5484] T c2d33db64ee8482aad840af219d60b8b P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: term changed from 0 to 1, leader changed from <none> to f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8). New cstate: current_term: 1 leader_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:12:58.438926 29731 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:12:58.445188 32641 raft_consensus.cc:1261] T def0775bc6eb4ae8b185ef113c276955 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:58.445868 567 raft_consensus.cc:1261] T def0775bc6eb4ae8b185ef113c276955 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:58.446589 643 consensus_queue.cc:1035] T def0775bc6eb4ae8b185ef113c276955 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:58.446827 709 consensus_queue.cc:1035] T def0775bc6eb4ae8b185ef113c276955 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:58.464187 824 consensus_queue.cc:1035] T 4cc7861a21f94dabb01133bac9ba0b46 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:58.471597 29725 catalog_manager.cc:5576] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595) does not have the latest schema for tablet def0775bc6eb4ae8b185ef113c276955 (table rebalancer_dynamic_table [id=59b3767ebde244f981465e9656416610]). Expected version 1 got 0
I20240503 02:12:58.472524 900 tablet.cc:1722] T def0775bc6eb4ae8b185ef113c276955 P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:58.473352 900 tablet.cc:1620] T def0775bc6eb4ae8b185ef113c276955 P 8a666e798fe8477d8dd1e51b385d5ead: MemRowSet was empty: no flush needed.
I20240503 02:12:58.487982 967 tablet.cc:1722] T def0775bc6eb4ae8b185ef113c276955 P 9a4d92bc25374fb3960fde7d10be309d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:58.490056 967 tablet.cc:1620] T def0775bc6eb4ae8b185ef113c276955 P 9a4d92bc25374fb3960fde7d10be309d: MemRowSet was empty: no flush needed.
I20240503 02:12:58.491149 961 mvcc.cc:205] Tried to move back new op lower bound from 7023420942105174016 to 7023420941638193152. Current Snapshot: MvccSnapshot[applied={T|T < 7023420942105174016}]
I20240503 02:12:58.493340 29731 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=59b3767ebde244f981465e9656416610] alter complete (version 1)
I20240503 02:12:58.493953 29734 catalog_manager.cc:5576] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233) does not have the latest schema for tablet def0775bc6eb4ae8b185ef113c276955 (table rebalancer_dynamic_table [id=59b3767ebde244f981465e9656416610]). Expected version 1 got 0
I20240503 02:12:58.498260 899 tablet.cc:1722] T def0775bc6eb4ae8b185ef113c276955 P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:58.498812 899 tablet.cc:1620] T def0775bc6eb4ae8b185ef113c276955 P f575ce92c0224eeb80b9fae5f6561fbe: MemRowSet was empty: no flush needed.
I20240503 02:12:58.499464 692 consensus_queue.cc:1035] T 4cc7861a21f94dabb01133bac9ba0b46 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:58.519085 29853 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:12:58.519932 709 consensus_queue.cc:1035] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:58.531701 29729 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240503 02:12:58.532150 29729 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240503 02:12:58.537626 29729 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet ad7b19f47b464333ad4c2736019c8fd6
I20240503 02:12:58.538671 29986 tablet_service.cc:1514] Processing DeleteTablet for tablet ad7b19f47b464333ad4c2736019c8fd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:58 UTC) from {username='slave'} at 127.0.0.1:52158
I20240503 02:12:58.539137 657 tablet_replica.cc:330] T ad7b19f47b464333ad4c2736019c8fd6 P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:12:58.539297 29729 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet aef29abf9b4e4cc195ad65ae15b00fd5
I20240503 02:12:58.539369 657 raft_consensus.cc:2229] T ad7b19f47b464333ad4c2736019c8fd6 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:58.539350 30244 tablet_service.cc:1514] Processing DeleteTablet for tablet ad7b19f47b464333ad4c2736019c8fd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:58 UTC) from {username='slave'} at 127.0.0.1:49856
I20240503 02:12:58.539664 657 raft_consensus.cc:2258] T ad7b19f47b464333ad4c2736019c8fd6 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:58.539598 643 consensus_queue.cc:1035] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:58.540290 29729 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet cb7965d0d50944c6be98c27f40f46de7
I20240503 02:12:58.540352 30244 tablet_service.cc:1514] Processing DeleteTablet for tablet aef29abf9b4e4cc195ad65ae15b00fd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:58 UTC) from {username='slave'} at 127.0.0.1:49856
I20240503 02:12:58.540542 657 ts_tablet_manager.cc:1875] T ad7b19f47b464333ad4c2736019c8fd6 P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:58.540649 30114 tablet_service.cc:1514] Processing DeleteTablet for tablet aef29abf9b4e4cc195ad65ae15b00fd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:58 UTC) from {username='slave'} at 127.0.0.1:44812
I20240503 02:12:58.540964 796 tablet_replica.cc:330] T aef29abf9b4e4cc195ad65ae15b00fd5 P a9094240860e43bfb129193f3bfd5be6: stopping tablet replica
I20240503 02:12:58.541173 796 raft_consensus.cc:2229] T aef29abf9b4e4cc195ad65ae15b00fd5 P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:58.541510 796 raft_consensus.cc:2258] T aef29abf9b4e4cc195ad65ae15b00fd5 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:58.542230 796 ts_tablet_manager.cc:1875] T aef29abf9b4e4cc195ad65ae15b00fd5 P a9094240860e43bfb129193f3bfd5be6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:58.545303 30244 tablet_service.cc:1514] Processing DeleteTablet for tablet cb7965d0d50944c6be98c27f40f46de7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:58 UTC) from {username='slave'} at 127.0.0.1:49856
I20240503 02:12:58.545434 656 tablet_replica.cc:330] T ad7b19f47b464333ad4c2736019c8fd6 P 1f10f80f13c84e80b03eda7085a3b012: stopping tablet replica
I20240503 02:12:58.545634 656 raft_consensus.cc:2229] T ad7b19f47b464333ad4c2736019c8fd6 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:58.545975 656 raft_consensus.cc:2258] T ad7b19f47b464333ad4c2736019c8fd6 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:58.546716 656 ts_tablet_manager.cc:1875] T ad7b19f47b464333ad4c2736019c8fd6 P 1f10f80f13c84e80b03eda7085a3b012: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:58.549546 796 ts_tablet_manager.cc:1888] T aef29abf9b4e4cc195ad65ae15b00fd5 P a9094240860e43bfb129193f3bfd5be6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:58.549665 796 log.cc:1196] T aef29abf9b4e4cc195ad65ae15b00fd5 P a9094240860e43bfb129193f3bfd5be6: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/wals/aef29abf9b4e4cc195ad65ae15b00fd5
I20240503 02:12:58.550093 796 ts_tablet_manager.cc:1909] T aef29abf9b4e4cc195ad65ae15b00fd5 P a9094240860e43bfb129193f3bfd5be6: Deleting consensus metadata
I20240503 02:12:58.550608 29984 tablet_service.cc:1514] Processing DeleteTablet for tablet cb7965d0d50944c6be98c27f40f46de7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:58 UTC) from {username='slave'} at 127.0.0.1:52158
I20240503 02:12:58.550910 29986 tablet_service.cc:1514] Processing DeleteTablet for tablet aef29abf9b4e4cc195ad65ae15b00fd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:58 UTC) from {username='slave'} at 127.0.0.1:52158
I20240503 02:12:58.551326 29718 catalog_manager.cc:4830] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): tablet aef29abf9b4e4cc195ad65ae15b00fd5 (table rebalance_test_table_10 [id=60b67e8bde6c4177a72606ad2be64acf]) successfully deleted
I20240503 02:12:58.558375 29853 tablet_service.cc:1514] Processing DeleteTablet for tablet ad7b19f47b464333ad4c2736019c8fd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:58 UTC) from {username='slave'} at 127.0.0.1:58744
I20240503 02:12:58.558331 29854 tablet_service.cc:1514] Processing DeleteTablet for tablet cb7965d0d50944c6be98c27f40f46de7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:58 UTC) from {username='slave'} at 127.0.0.1:58744
I20240503 02:12:58.584419 656 ts_tablet_manager.cc:1888] T ad7b19f47b464333ad4c2736019c8fd6 P 1f10f80f13c84e80b03eda7085a3b012: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:58.584646 656 log.cc:1196] T ad7b19f47b464333ad4c2736019c8fd6 P 1f10f80f13c84e80b03eda7085a3b012: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/wals/ad7b19f47b464333ad4c2736019c8fd6
I20240503 02:12:58.585227 656 ts_tablet_manager.cc:1909] T ad7b19f47b464333ad4c2736019c8fd6 P 1f10f80f13c84e80b03eda7085a3b012: Deleting consensus metadata
I20240503 02:12:58.590142 657 ts_tablet_manager.cc:1888] T ad7b19f47b464333ad4c2736019c8fd6 P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:58.590344 657 log.cc:1196] T ad7b19f47b464333ad4c2736019c8fd6 P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/ad7b19f47b464333ad4c2736019c8fd6
I20240503 02:12:58.590881 657 ts_tablet_manager.cc:1909] T ad7b19f47b464333ad4c2736019c8fd6 P 155735c7f8b14997ac4c770aee2a9401: Deleting consensus metadata
I20240503 02:12:58.592339 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet ad7b19f47b464333ad4c2736019c8fd6 (table rebalance_test_table_10 [id=60b67e8bde6c4177a72606ad2be64acf]) successfully deleted
I20240503 02:12:58.598109 29729 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:12:58.614465 900 tablet.cc:1722] T def0775bc6eb4ae8b185ef113c276955 P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:12:58.615239 900 tablet.cc:1620] T def0775bc6eb4ae8b185ef113c276955 P 8a666e798fe8477d8dd1e51b385d5ead: MemRowSet was empty: no flush needed.
I20240503 02:12:58.615793 899 tablet.cc:1722] T def0775bc6eb4ae8b185ef113c276955 P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:12:58.616299 899 tablet.cc:1620] T def0775bc6eb4ae8b185ef113c276955 P f575ce92c0224eeb80b9fae5f6561fbe: MemRowSet was empty: no flush needed.
I20240503 02:12:58.621448 29986 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:12:58.631086 29719 catalog_manager.cc:4830] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): tablet ad7b19f47b464333ad4c2736019c8fd6 (table rebalance_test_table_10 [id=60b67e8bde6c4177a72606ad2be64acf]) successfully deleted
I20240503 02:12:58.631731 656 tablet_replica.cc:330] T aef29abf9b4e4cc195ad65ae15b00fd5 P 1f10f80f13c84e80b03eda7085a3b012: stopping tablet replica
I20240503 02:12:58.631948 656 raft_consensus.cc:2229] T aef29abf9b4e4cc195ad65ae15b00fd5 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:58.632272 656 raft_consensus.cc:2258] T aef29abf9b4e4cc195ad65ae15b00fd5 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:58.633677 967 tablet.cc:1722] T def0775bc6eb4ae8b185ef113c276955 P 9a4d92bc25374fb3960fde7d10be309d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:12:58.633906 980 tablet_replica.cc:330] T ad7b19f47b464333ad4c2736019c8fd6 P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:12:58.634207 967 tablet.cc:1620] T def0775bc6eb4ae8b185ef113c276955 P 9a4d92bc25374fb3960fde7d10be309d: MemRowSet was empty: no flush needed.
I20240503 02:12:58.637318 29729 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=59b3767ebde244f981465e9656416610] alter complete (version 2)
I20240503 02:12:58.646051 657 tablet_replica.cc:330] T cb7965d0d50944c6be98c27f40f46de7 P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:12:58.647755 657 raft_consensus.cc:2229] T cb7965d0d50944c6be98c27f40f46de7 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:58.647907 32754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:58.648087 30244 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:12:58.648173 657 raft_consensus.cc:2258] T cb7965d0d50944c6be98c27f40f46de7 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:58.636493 980 raft_consensus.cc:2229] T ad7b19f47b464333ad4c2736019c8fd6 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:58.654428 32620 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:58.654839 980 raft_consensus.cc:2258] T ad7b19f47b464333ad4c2736019c8fd6 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:58.656093 980 ts_tablet_manager.cc:1875] T ad7b19f47b464333ad4c2736019c8fd6 P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:58.656785 657 ts_tablet_manager.cc:1875] T cb7965d0d50944c6be98c27f40f46de7 P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:58.666582 656 ts_tablet_manager.cc:1875] T aef29abf9b4e4cc195ad65ae15b00fd5 P 1f10f80f13c84e80b03eda7085a3b012: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:58.685276 980 ts_tablet_manager.cc:1888] T ad7b19f47b464333ad4c2736019c8fd6 P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:58.688019 657 ts_tablet_manager.cc:1888] T cb7965d0d50944c6be98c27f40f46de7 P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:58.688176 657 log.cc:1196] T cb7965d0d50944c6be98c27f40f46de7 P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/cb7965d0d50944c6be98c27f40f46de7
I20240503 02:12:58.688326 980 log.cc:1196] T ad7b19f47b464333ad4c2736019c8fd6 P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/ad7b19f47b464333ad4c2736019c8fd6
I20240503 02:12:58.688660 657 ts_tablet_manager.cc:1909] T cb7965d0d50944c6be98c27f40f46de7 P 155735c7f8b14997ac4c770aee2a9401: Deleting consensus metadata
I20240503 02:12:58.690629 980 ts_tablet_manager.cc:1909] T ad7b19f47b464333ad4c2736019c8fd6 P 16440e365430405fb10366ab70e2b429: Deleting consensus metadata
I20240503 02:12:58.691673 656 ts_tablet_manager.cc:1888] T aef29abf9b4e4cc195ad65ae15b00fd5 P 1f10f80f13c84e80b03eda7085a3b012: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:58.691813 656 log.cc:1196] T aef29abf9b4e4cc195ad65ae15b00fd5 P 1f10f80f13c84e80b03eda7085a3b012: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/wals/aef29abf9b4e4cc195ad65ae15b00fd5
I20240503 02:12:58.692235 656 ts_tablet_manager.cc:1909] T aef29abf9b4e4cc195ad65ae15b00fd5 P 1f10f80f13c84e80b03eda7085a3b012: Deleting consensus metadata
I20240503 02:12:58.692267 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet ad7b19f47b464333ad4c2736019c8fd6 (table rebalance_test_table_10 [id=60b67e8bde6c4177a72606ad2be64acf]) successfully deleted
I20240503 02:12:58.692854 980 tablet_replica.cc:330] T cb7965d0d50944c6be98c27f40f46de7 P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:12:58.693224 980 raft_consensus.cc:2229] T cb7965d0d50944c6be98c27f40f46de7 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:58.693586 980 raft_consensus.cc:2258] T cb7965d0d50944c6be98c27f40f46de7 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:58.694694 980 ts_tablet_manager.cc:1875] T cb7965d0d50944c6be98c27f40f46de7 P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:58.696631 656 tablet_replica.cc:330] T cb7965d0d50944c6be98c27f40f46de7 P 1f10f80f13c84e80b03eda7085a3b012: stopping tablet replica
I20240503 02:12:58.696825 29719 catalog_manager.cc:4830] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): tablet aef29abf9b4e4cc195ad65ae15b00fd5 (table rebalance_test_table_10 [id=60b67e8bde6c4177a72606ad2be64acf]) successfully deleted
I20240503 02:12:58.696875 656 raft_consensus.cc:2229] T cb7965d0d50944c6be98c27f40f46de7 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:58.697130 656 raft_consensus.cc:2258] T cb7965d0d50944c6be98c27f40f46de7 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:58.697906 656 ts_tablet_manager.cc:1875] T cb7965d0d50944c6be98c27f40f46de7 P 1f10f80f13c84e80b03eda7085a3b012: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:58.711103 656 ts_tablet_manager.cc:1888] T cb7965d0d50944c6be98c27f40f46de7 P 1f10f80f13c84e80b03eda7085a3b012: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:58.711278 656 log.cc:1196] T cb7965d0d50944c6be98c27f40f46de7 P 1f10f80f13c84e80b03eda7085a3b012: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/wals/cb7965d0d50944c6be98c27f40f46de7
I20240503 02:12:58.711735 656 ts_tablet_manager.cc:1909] T cb7965d0d50944c6be98c27f40f46de7 P 1f10f80f13c84e80b03eda7085a3b012: Deleting consensus metadata
I20240503 02:12:58.714982 29719 catalog_manager.cc:4830] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): tablet cb7965d0d50944c6be98c27f40f46de7 (table rebalance_test_table_10 [id=60b67e8bde6c4177a72606ad2be64acf]) successfully deleted
I20240503 02:12:58.723316 980 ts_tablet_manager.cc:1888] T cb7965d0d50944c6be98c27f40f46de7 P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:58.729213 980 log.cc:1196] T cb7965d0d50944c6be98c27f40f46de7 P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/cb7965d0d50944c6be98c27f40f46de7
I20240503 02:12:58.729838 980 ts_tablet_manager.cc:1909] T cb7965d0d50944c6be98c27f40f46de7 P 16440e365430405fb10366ab70e2b429: Deleting consensus metadata
I20240503 02:12:58.731614 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet cb7965d0d50944c6be98c27f40f46de7 (table rebalance_test_table_10 [id=60b67e8bde6c4177a72606ad2be64acf]) successfully deleted
I20240503 02:12:58.742619 30114 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:12:58.743332 29734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:12:58.743620 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet cb7965d0d50944c6be98c27f40f46de7 (table rebalance_test_table_10 [id=60b67e8bde6c4177a72606ad2be64acf]) successfully deleted
I20240503 02:12:58.743839 546 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:58.744081 657 tablet_replica.cc:330] T aef29abf9b4e4cc195ad65ae15b00fd5 P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:12:58.744307 657 raft_consensus.cc:2229] T aef29abf9b4e4cc195ad65ae15b00fd5 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:58.744570 657 raft_consensus.cc:2258] T aef29abf9b4e4cc195ad65ae15b00fd5 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:58.748209 417 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:58.749622 657 ts_tablet_manager.cc:1875] T aef29abf9b4e4cc195ad65ae15b00fd5 P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:58.756354 657 ts_tablet_manager.cc:1888] T aef29abf9b4e4cc195ad65ae15b00fd5 P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:58.756534 657 log.cc:1196] T aef29abf9b4e4cc195ad65ae15b00fd5 P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/aef29abf9b4e4cc195ad65ae15b00fd5
I20240503 02:12:58.757280 900 tablet.cc:1722] T def0775bc6eb4ae8b185ef113c276955 P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:12:58.764664 899 tablet.cc:1722] T def0775bc6eb4ae8b185ef113c276955 P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:12:58.771518 967 tablet.cc:1722] T def0775bc6eb4ae8b185ef113c276955 P 9a4d92bc25374fb3960fde7d10be309d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:12:58.757007 657 ts_tablet_manager.cc:1909] T aef29abf9b4e4cc195ad65ae15b00fd5 P 155735c7f8b14997ac4c770aee2a9401: Deleting consensus metadata
I20240503 02:12:58.774403 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet aef29abf9b4e4cc195ad65ae15b00fd5 (table rebalance_test_table_10 [id=60b67e8bde6c4177a72606ad2be64acf]) successfully deleted
I20240503 02:12:58.784165 29734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table_new_name [id=59b3767ebde244f981465e9656416610] alter complete (version 3)
I20240503 02:12:58.811249 877 consensus_queue.cc:1035] T c2d33db64ee8482aad840af219d60b8b P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:58.842602 876 consensus_queue.cc:1035] T c2d33db64ee8482aad840af219d60b8b P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:58.894330 29729 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:12:58.894552 29729 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:12:58.898164 29729 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet def0775bc6eb4ae8b185ef113c276955
I20240503 02:12:58.899039 546 tablet_service.cc:1514] Processing DeleteTablet for tablet def0775bc6eb4ae8b185ef113c276955 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:58 UTC) from {username='slave'} at 127.0.0.1:39028
I20240503 02:12:58.900506 32754 tablet_service.cc:1514] Processing DeleteTablet for tablet def0775bc6eb4ae8b185ef113c276955 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:58 UTC) from {username='slave'} at 127.0.0.1:46360
I20240503 02:12:58.900748 32620 tablet_service.cc:1514] Processing DeleteTablet for tablet def0775bc6eb4ae8b185ef113c276955 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:58 UTC) from {username='slave'} at 127.0.0.1:38200
I20240503 02:12:58.905395 993 tablet_replica.cc:330] T def0775bc6eb4ae8b185ef113c276955 P 8a666e798fe8477d8dd1e51b385d5ead: stopping tablet replica
I20240503 02:12:58.905779 993 raft_consensus.cc:2229] T def0775bc6eb4ae8b185ef113c276955 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:58.906303 993 raft_consensus.cc:2258] T def0775bc6eb4ae8b185ef113c276955 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:58.907536 993 ts_tablet_manager.cc:1875] T def0775bc6eb4ae8b185ef113c276955 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:58.912164 994 tablet_replica.cc:330] T def0775bc6eb4ae8b185ef113c276955 P 9a4d92bc25374fb3960fde7d10be309d: stopping tablet replica
I20240503 02:12:58.912947 994 raft_consensus.cc:2229] T def0775bc6eb4ae8b185ef113c276955 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:58.914316 994 raft_consensus.cc:2258] T def0775bc6eb4ae8b185ef113c276955 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:58.917281 994 ts_tablet_manager.cc:1875] T def0775bc6eb4ae8b185ef113c276955 P 9a4d92bc25374fb3960fde7d10be309d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:58.925465 993 ts_tablet_manager.cc:1888] T def0775bc6eb4ae8b185ef113c276955 P 8a666e798fe8477d8dd1e51b385d5ead: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:58.927271 993 log.cc:1196] T def0775bc6eb4ae8b185ef113c276955 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal/wals/def0775bc6eb4ae8b185ef113c276955
I20240503 02:12:58.927884 993 ts_tablet_manager.cc:1909] T def0775bc6eb4ae8b185ef113c276955 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting consensus metadata
I20240503 02:12:58.931519 994 ts_tablet_manager.cc:1888] T def0775bc6eb4ae8b185ef113c276955 P 9a4d92bc25374fb3960fde7d10be309d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:58.931932 992 tablet_replica.cc:330] T def0775bc6eb4ae8b185ef113c276955 P f575ce92c0224eeb80b9fae5f6561fbe: stopping tablet replica
I20240503 02:12:58.932219 992 raft_consensus.cc:2229] T def0775bc6eb4ae8b185ef113c276955 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:58.932519 992 raft_consensus.cc:2258] T def0775bc6eb4ae8b185ef113c276955 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:58.932794 29719 catalog_manager.cc:4830] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): tablet def0775bc6eb4ae8b185ef113c276955 (table rebalancer_dynamic_table_new_name [id=59b3767ebde244f981465e9656416610]) successfully deleted
I20240503 02:12:58.933156 994 log.cc:1196] T def0775bc6eb4ae8b185ef113c276955 P 9a4d92bc25374fb3960fde7d10be309d: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal/wals/def0775bc6eb4ae8b185ef113c276955
I20240503 02:12:58.933846 992 ts_tablet_manager.cc:1875] T def0775bc6eb4ae8b185ef113c276955 P f575ce92c0224eeb80b9fae5f6561fbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:58.933871 994 ts_tablet_manager.cc:1909] T def0775bc6eb4ae8b185ef113c276955 P 9a4d92bc25374fb3960fde7d10be309d: Deleting consensus metadata
I20240503 02:12:58.935716 29719 catalog_manager.cc:4830] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): tablet def0775bc6eb4ae8b185ef113c276955 (table rebalancer_dynamic_table_new_name [id=59b3767ebde244f981465e9656416610]) successfully deleted
I20240503 02:12:58.942620 992 ts_tablet_manager.cc:1888] T def0775bc6eb4ae8b185ef113c276955 P f575ce92c0224eeb80b9fae5f6561fbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:58.942817 992 log.cc:1196] T def0775bc6eb4ae8b185ef113c276955 P f575ce92c0224eeb80b9fae5f6561fbe: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal/wals/def0775bc6eb4ae8b185ef113c276955
I20240503 02:12:58.943322 992 ts_tablet_manager.cc:1909] T def0775bc6eb4ae8b185ef113c276955 P f575ce92c0224eeb80b9fae5f6561fbe: Deleting consensus metadata
I20240503 02:12:58.946198 29718 catalog_manager.cc:4830] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): tablet def0775bc6eb4ae8b185ef113c276955 (table rebalancer_dynamic_table_new_name [id=59b3767ebde244f981465e9656416610]) successfully deleted
I20240503 02:12:59.002277 30004 consensus_queue.cc:237] T a592cdf6d1d747c68b5334b90006fa8b P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:12:59.002466 29729 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:12:59.005409 29873 raft_consensus.cc:1261] T a592cdf6d1d747c68b5334b90006fa8b P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:12:59.006106 30134 raft_consensus.cc:1261] T a592cdf6d1d747c68b5334b90006fa8b P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:12:59.006258 638 consensus_queue.cc:1035] T a592cdf6d1d747c68b5334b90006fa8b P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:12:59.007018 638 consensus_queue.cc:1035] T a592cdf6d1d747c68b5334b90006fa8b P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:12:59.011767 29986 tablet_service.cc:1467] Processing CreateTablet for tablet 33734f77af5342af82ee295bcf71f85a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4c683f284ec140f69703e915a553a141]), partition=
I20240503 02:12:59.012236 29986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33734f77af5342af82ee295bcf71f85a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.017946 546 tablet_service.cc:1467] Processing CreateTablet for tablet 33734f77af5342af82ee295bcf71f85a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4c683f284ec140f69703e915a553a141]), partition=
I20240503 02:12:59.018409 546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33734f77af5342af82ee295bcf71f85a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.021791 417 tablet_service.cc:1467] Processing CreateTablet for tablet 33734f77af5342af82ee295bcf71f85a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4c683f284ec140f69703e915a553a141]), partition=
I20240503 02:12:59.022164 417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33734f77af5342af82ee295bcf71f85a. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:59.025966 29940 consensus_peers.cc:485] T a592cdf6d1d747c68b5334b90006fa8b P 155735c7f8b14997ac4c770aee2a9401 -> Peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): Couldn't send request to peer 8a666e798fe8477d8dd1e51b385d5ead. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a592cdf6d1d747c68b5334b90006fa8b. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:59.029105 1015 tablet_bootstrap.cc:492] T 33734f77af5342af82ee295bcf71f85a P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:12:59.032768 1015 tablet_bootstrap.cc:654] T 33734f77af5342af82ee295bcf71f85a P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:59.034710 1016 tablet_bootstrap.cc:492] T 33734f77af5342af82ee295bcf71f85a P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:59.035946 1017 tablet_bootstrap.cc:492] T 33734f77af5342af82ee295bcf71f85a P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:12:59.036288 1015 tablet_bootstrap.cc:492] T 33734f77af5342af82ee295bcf71f85a P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:12:59.036484 1015 ts_tablet_manager.cc:1367] T 33734f77af5342af82ee295bcf71f85a P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240503 02:12:59.036408 638 raft_consensus.cc:2932] T a592cdf6d1d747c68b5334b90006fa8b P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:12:59.037305 1015 raft_consensus.cc:348] T 33734f77af5342af82ee295bcf71f85a P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:59.037515 1015 raft_consensus.cc:374] T 33734f77af5342af82ee295bcf71f85a P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:59.037621 1015 raft_consensus.cc:729] T 33734f77af5342af82ee295bcf71f85a P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:12:59.037854 1017 tablet_bootstrap.cc:654] T 33734f77af5342af82ee295bcf71f85a P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:59.037861 1015 consensus_queue.cc:260] T 33734f77af5342af82ee295bcf71f85a P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:59.038380 1015 ts_tablet_manager.cc:1398] T 33734f77af5342af82ee295bcf71f85a P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:59.039448 30134 raft_consensus.cc:2932] T a592cdf6d1d747c68b5334b90006fa8b P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:12:59.046906 1016 tablet_bootstrap.cc:654] T 33734f77af5342af82ee295bcf71f85a P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:59.049881 29729 catalog_manager.cc:5484] T a592cdf6d1d747c68b5334b90006fa8b P a9094240860e43bfb129193f3bfd5be6 reported cstate change: config changed from index -1 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:12:59.049993 29873 raft_consensus.cc:2932] T a592cdf6d1d747c68b5334b90006fa8b P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:12:59.053705 1017 tablet_bootstrap.cc:492] T 33734f77af5342af82ee295bcf71f85a P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:12:59.053897 1017 ts_tablet_manager.cc:1367] T 33734f77af5342af82ee295bcf71f85a P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.008s
I20240503 02:12:59.054414 29734 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_extra_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:59.054692 1017 raft_consensus.cc:348] T 33734f77af5342af82ee295bcf71f85a P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:59.054900 1017 raft_consensus.cc:374] T 33734f77af5342af82ee295bcf71f85a P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:59.054987 1017 raft_consensus.cc:729] T 33734f77af5342af82ee295bcf71f85a P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:12:59.055229 1017 consensus_queue.cc:260] T 33734f77af5342af82ee295bcf71f85a P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:59.056735 1017 ts_tablet_manager.cc:1398] T 33734f77af5342af82ee295bcf71f85a P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:12:59.058672 29729 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240503 02:12:59.058892 29729 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240503 02:12:59.059155 1016 tablet_bootstrap.cc:492] T 33734f77af5342af82ee295bcf71f85a P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:59.059301 1016 ts_tablet_manager.cc:1367] T 33734f77af5342af82ee295bcf71f85a P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.006s
I20240503 02:12:59.060151 1016 raft_consensus.cc:348] T 33734f77af5342af82ee295bcf71f85a P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:59.060468 1016 raft_consensus.cc:374] T 33734f77af5342af82ee295bcf71f85a P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:59.060619 1016 raft_consensus.cc:729] T 33734f77af5342af82ee295bcf71f85a P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:59.060914 1016 consensus_queue.cc:260] T 33734f77af5342af82ee295bcf71f85a P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:59.063010 1016 ts_tablet_manager.cc:1398] T 33734f77af5342af82ee295bcf71f85a P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:12:59.067646 29729 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 4 replicas of tablet a592cdf6d1d747c68b5334b90006fa8b
I20240503 02:12:59.068343 29729 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet 445caf2423c1488b8e47474032e68488
I20240503 02:12:59.068626 29853 tablet_service.cc:1514] Processing DeleteTablet for tablet a592cdf6d1d747c68b5334b90006fa8b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:59 UTC) from {username='slave'} at 127.0.0.1:58744
I20240503 02:12:59.069103 29854 tablet_service.cc:1467] Processing CreateTablet for tablet 70b8ebcf6f3245e6bd93eeb38ca1cbf5 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=53af41732fcb40aea2f1c6f0ac294bde]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:59.069564 980 tablet_replica.cc:330] T a592cdf6d1d747c68b5334b90006fa8b P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:12:59.069720 29854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70b8ebcf6f3245e6bd93eeb38ca1cbf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.069665 29986 tablet_service.cc:1467] Processing CreateTablet for tablet 70b8ebcf6f3245e6bd93eeb38ca1cbf5 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=53af41732fcb40aea2f1c6f0ac294bde]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:59.069784 980 raft_consensus.cc:2229] T a592cdf6d1d747c68b5334b90006fa8b P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:59.070070 29986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70b8ebcf6f3245e6bd93eeb38ca1cbf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.070129 980 raft_consensus.cc:2258] T a592cdf6d1d747c68b5334b90006fa8b P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:59.070426 29984 tablet_service.cc:1514] Processing DeleteTablet for tablet a592cdf6d1d747c68b5334b90006fa8b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:59 UTC) from {username='slave'} at 127.0.0.1:52158
I20240503 02:12:59.070721 657 tablet_replica.cc:330] T a592cdf6d1d747c68b5334b90006fa8b P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:12:59.070945 657 raft_consensus.cc:2229] T a592cdf6d1d747c68b5334b90006fa8b P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:59.071374 30114 tablet_service.cc:1514] Processing DeleteTablet for tablet a592cdf6d1d747c68b5334b90006fa8b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:59 UTC) from {username='slave'} at 127.0.0.1:44812
I20240503 02:12:59.071472 657 raft_consensus.cc:2258] T a592cdf6d1d747c68b5334b90006fa8b P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:59.073760 29853 tablet_service.cc:1514] Processing DeleteTablet for tablet 445caf2423c1488b8e47474032e68488 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:59 UTC) from {username='slave'} at 127.0.0.1:58744
I20240503 02:12:59.073863 30244 tablet_service.cc:1514] Processing DeleteTablet for tablet 445caf2423c1488b8e47474032e68488 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:59 UTC) from {username='slave'} at 127.0.0.1:49856
I20240503 02:12:59.074038 32754 tablet_service.cc:1514] Processing DeleteTablet for tablet a592cdf6d1d747c68b5334b90006fa8b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:59 UTC) from {username='slave'} at 127.0.0.1:46360
I20240503 02:12:59.074223 656 tablet_replica.cc:330] T 445caf2423c1488b8e47474032e68488 P 1f10f80f13c84e80b03eda7085a3b012: stopping tablet replica
I20240503 02:12:59.074412 656 raft_consensus.cc:2229] T 445caf2423c1488b8e47474032e68488 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:59.074527 657 ts_tablet_manager.cc:1875] T a592cdf6d1d747c68b5334b90006fa8b P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:59.074776 656 raft_consensus.cc:2258] T 445caf2423c1488b8e47474032e68488 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:59.074721 32753 tablet_service.cc:1467] Processing CreateTablet for tablet 8f5e6ff35b994d46a1e1fe216eb356ce (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=53af41732fcb40aea2f1c6f0ac294bde]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:59.075239 32753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f5e6ff35b994d46a1e1fe216eb356ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.075470 656 ts_tablet_manager.cc:1875] T 445caf2423c1488b8e47474032e68488 P 1f10f80f13c84e80b03eda7085a3b012: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:59.077241 29984 tablet_service.cc:1514] Processing DeleteTablet for tablet 445caf2423c1488b8e47474032e68488 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:59 UTC) from {username='slave'} at 127.0.0.1:52158
W20240503 02:12:59.078783 29719 catalog_manager.cc:4797] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): delete failed for tablet a592cdf6d1d747c68b5334b90006fa8b because the tablet was not found. No further retry: Not found: Tablet not found: a592cdf6d1d747c68b5334b90006fa8b
I20240503 02:12:59.079353 980 ts_tablet_manager.cc:1875] T a592cdf6d1d747c68b5334b90006fa8b P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:59.084731 30244 tablet_service.cc:1467] Processing CreateTablet for tablet 8f5e6ff35b994d46a1e1fe216eb356ce (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=53af41732fcb40aea2f1c6f0ac294bde]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:59.085199 30244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f5e6ff35b994d46a1e1fe216eb356ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.086748 547 tablet_service.cc:1467] Processing CreateTablet for tablet ba97049157a646c780fa203d34cd4c82 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=53af41732fcb40aea2f1c6f0ac294bde]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:59.086975 417 tablet_service.cc:1467] Processing CreateTablet for tablet 8f5e6ff35b994d46a1e1fe216eb356ce (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=53af41732fcb40aea2f1c6f0ac294bde]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:59.087258 547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba97049157a646c780fa203d34cd4c82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.087383 417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f5e6ff35b994d46a1e1fe216eb356ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.089382 416 tablet_service.cc:1467] Processing CreateTablet for tablet ba97049157a646c780fa203d34cd4c82 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=53af41732fcb40aea2f1c6f0ac294bde]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:59.089735 416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba97049157a646c780fa203d34cd4c82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.094983 1016 tablet_bootstrap.cc:492] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:59.095330 30245 tablet_service.cc:1467] Processing CreateTablet for tablet ba97049157a646c780fa203d34cd4c82 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=53af41732fcb40aea2f1c6f0ac294bde]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:59.095726 30245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba97049157a646c780fa203d34cd4c82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.096594 1016 tablet_bootstrap.cc:654] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:59.100602 1015 tablet_bootstrap.cc:492] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:12:59.102159 1015 tablet_bootstrap.cc:654] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:59.117362 1028 tablet_bootstrap.cc:492] T 8f5e6ff35b994d46a1e1fe216eb356ce P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:12:59.121140 1028 tablet_bootstrap.cc:654] T 8f5e6ff35b994d46a1e1fe216eb356ce P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:59.121196 546 tablet_service.cc:1467] Processing CreateTablet for tablet 70b8ebcf6f3245e6bd93eeb38ca1cbf5 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=53af41732fcb40aea2f1c6f0ac294bde]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:59.121704 546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70b8ebcf6f3245e6bd93eeb38ca1cbf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.121829 1029 tablet_bootstrap.cc:492] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:12:59.123713 657 ts_tablet_manager.cc:1888] T a592cdf6d1d747c68b5334b90006fa8b P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:59.123919 657 log.cc:1196] T a592cdf6d1d747c68b5334b90006fa8b P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/a592cdf6d1d747c68b5334b90006fa8b
I20240503 02:12:59.124378 657 ts_tablet_manager.cc:1909] T a592cdf6d1d747c68b5334b90006fa8b P 155735c7f8b14997ac4c770aee2a9401: Deleting consensus metadata
I20240503 02:12:59.125483 1029 tablet_bootstrap.cc:654] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:59.125764 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet a592cdf6d1d747c68b5334b90006fa8b (table rebalance_test_table_12 [id=c501c7cdd0d142858bc57dffeae345f8]) successfully deleted
I20240503 02:12:59.126098 29729 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet a74d38f7d6e5441396944c52acd41c74
I20240503 02:12:59.126965 30116 tablet_service.cc:1514] Processing DeleteTablet for tablet a74d38f7d6e5441396944c52acd41c74 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:59 UTC) from {username='slave'} at 127.0.0.1:44812
I20240503 02:12:59.127899 30246 tablet_service.cc:1514] Processing DeleteTablet for tablet a74d38f7d6e5441396944c52acd41c74 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:59 UTC) from {username='slave'} at 127.0.0.1:49856
I20240503 02:12:59.126297 980 ts_tablet_manager.cc:1888] T a592cdf6d1d747c68b5334b90006fa8b P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:59.128182 980 log.cc:1196] T a592cdf6d1d747c68b5334b90006fa8b P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/a592cdf6d1d747c68b5334b90006fa8b
I20240503 02:12:59.129493 1015 tablet_bootstrap.cc:492] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:12:59.129679 1015 ts_tablet_manager.cc:1367] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.029s user 0.013s sys 0.000s
I20240503 02:12:59.130091 1016 tablet_bootstrap.cc:492] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:12:59.130262 1016 ts_tablet_manager.cc:1367] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.035s user 0.000s sys 0.014s
I20240503 02:12:59.130344 1015 raft_consensus.cc:348] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:12:59.130561 1015 raft_consensus.cc:374] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:59.130645 1015 raft_consensus.cc:729] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:12:59.130934 1015 consensus_queue.cc:260] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:12:59.131006 1016 raft_consensus.cc:348] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:59.131228 1016 raft_consensus.cc:374] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:59.131426 1015 ts_tablet_manager.cc:1398] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:59.131750 1015 tablet_bootstrap.cc:492] T 8f5e6ff35b994d46a1e1fe216eb356ce P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:12:59.133086 29854 tablet_service.cc:1514] Processing DeleteTablet for tablet a74d38f7d6e5441396944c52acd41c74 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:59 UTC) from {username='slave'} at 127.0.0.1:58744
I20240503 02:12:59.133106 980 ts_tablet_manager.cc:1909] T a592cdf6d1d747c68b5334b90006fa8b P 16440e365430405fb10366ab70e2b429: Deleting consensus metadata
I20240503 02:12:59.133338 656 ts_tablet_manager.cc:1888] T 445caf2423c1488b8e47474032e68488 P 1f10f80f13c84e80b03eda7085a3b012: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:59.133495 656 log.cc:1196] T 445caf2423c1488b8e47474032e68488 P 1f10f80f13c84e80b03eda7085a3b012: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/wals/445caf2423c1488b8e47474032e68488
I20240503 02:12:59.133563 1017 tablet_bootstrap.cc:492] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:12:59.133955 656 ts_tablet_manager.cc:1909] T 445caf2423c1488b8e47474032e68488 P 1f10f80f13c84e80b03eda7085a3b012: Deleting consensus metadata
I20240503 02:12:59.134964 980 tablet_replica.cc:330] T 445caf2423c1488b8e47474032e68488 P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:12:59.135104 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet a592cdf6d1d747c68b5334b90006fa8b (table rebalance_test_table_12 [id=c501c7cdd0d142858bc57dffeae345f8]) successfully deleted
I20240503 02:12:59.135250 1017 tablet_bootstrap.cc:654] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:59.136054 29719 catalog_manager.cc:4830] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): tablet 445caf2423c1488b8e47474032e68488 (table rebalance_test_table_12 [id=c501c7cdd0d142858bc57dffeae345f8]) successfully deleted
I20240503 02:12:59.136440 980 raft_consensus.cc:2229] T 445caf2423c1488b8e47474032e68488 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:59.136514 656 tablet_replica.cc:330] T a74d38f7d6e5441396944c52acd41c74 P 1f10f80f13c84e80b03eda7085a3b012: stopping tablet replica
I20240503 02:12:59.136744 980 raft_consensus.cc:2258] T 445caf2423c1488b8e47474032e68488 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:59.136839 656 raft_consensus.cc:2229] T a74d38f7d6e5441396944c52acd41c74 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:59.137219 656 raft_consensus.cc:2258] T a74d38f7d6e5441396944c52acd41c74 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:59.137702 980 ts_tablet_manager.cc:1875] T 445caf2423c1488b8e47474032e68488 P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:59.138118 656 ts_tablet_manager.cc:1875] T a74d38f7d6e5441396944c52acd41c74 P 1f10f80f13c84e80b03eda7085a3b012: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:59.138166 1015 tablet_bootstrap.cc:654] T 8f5e6ff35b994d46a1e1fe216eb356ce P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:59.140128 1024 tablet_replica.cc:330] T a592cdf6d1d747c68b5334b90006fa8b P a9094240860e43bfb129193f3bfd5be6: stopping tablet replica
I20240503 02:12:59.140640 1024 raft_consensus.cc:2229] T a592cdf6d1d747c68b5334b90006fa8b P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:59.140671 1016 raft_consensus.cc:729] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:12:59.141307 1016 consensus_queue.cc:260] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:59.141633 1025 tablet_bootstrap.cc:492] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:59.142699 986 raft_consensus.cc:482] T 33734f77af5342af82ee295bcf71f85a P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:59.142834 1016 ts_tablet_manager.cc:1398] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20240503 02:12:59.142851 986 raft_consensus.cc:504] T 33734f77af5342af82ee295bcf71f85a P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:59.143627 1025 tablet_bootstrap.cc:654] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:59.144176 986 leader_election.cc:288] T 33734f77af5342af82ee295bcf71f85a P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:12:59.147374 1029 tablet_bootstrap.cc:492] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:12:59.147610 1029 ts_tablet_manager.cc:1367] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.002s
I20240503 02:12:59.148492 1029 raft_consensus.cc:348] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:59.149281 1029 raft_consensus.cc:374] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:59.149478 1029 raft_consensus.cc:729] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:12:59.149792 1029 consensus_queue.cc:260] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:59.150624 1029 ts_tablet_manager.cc:1398] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:12:59.135470 657 tablet_replica.cc:330] T 445caf2423c1488b8e47474032e68488 P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:12:59.151374 657 raft_consensus.cc:2229] T 445caf2423c1488b8e47474032e68488 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:59.151692 657 raft_consensus.cc:2258] T 445caf2423c1488b8e47474032e68488 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:59.155047 657 ts_tablet_manager.cc:1875] T 445caf2423c1488b8e47474032e68488 P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:59.157382 1024 raft_consensus.cc:2258] T a592cdf6d1d747c68b5334b90006fa8b P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:59.159619 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33734f77af5342af82ee295bcf71f85a" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:12:59.159904 437 raft_consensus.cc:2454] T 33734f77af5342af82ee295bcf71f85a P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:12:59.147730 1028 tablet_bootstrap.cc:492] T 8f5e6ff35b994d46a1e1fe216eb356ce P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:12:59.160367 29940 leader_election.cc:302] T 33734f77af5342af82ee295bcf71f85a P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:12:59.160573 1028 ts_tablet_manager.cc:1367] T 8f5e6ff35b994d46a1e1fe216eb356ce P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.043s user 0.000s sys 0.005s
I20240503 02:12:59.161121 981 raft_consensus.cc:2781] T 33734f77af5342af82ee295bcf71f85a P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:59.161270 981 raft_consensus.cc:482] T 33734f77af5342af82ee295bcf71f85a P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:59.161602 1015 tablet_bootstrap.cc:492] T 8f5e6ff35b994d46a1e1fe216eb356ce P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:12:59.161644 981 raft_consensus.cc:3037] T 33734f77af5342af82ee295bcf71f85a P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:59.161751 1015 ts_tablet_manager.cc:1367] T 8f5e6ff35b994d46a1e1fe216eb356ce P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20240503 02:12:59.162117 1028 raft_consensus.cc:348] T 8f5e6ff35b994d46a1e1fe216eb356ce P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:59.162544 1024 ts_tablet_manager.cc:1875] T a592cdf6d1d747c68b5334b90006fa8b P a9094240860e43bfb129193f3bfd5be6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:59.162536 1015 raft_consensus.cc:348] T 8f5e6ff35b994d46a1e1fe216eb356ce P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:59.162750 1015 raft_consensus.cc:374] T 8f5e6ff35b994d46a1e1fe216eb356ce P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:59.162863 1015 raft_consensus.cc:729] T 8f5e6ff35b994d46a1e1fe216eb356ce P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:12:59.163107 1015 consensus_queue.cc:260] T 8f5e6ff35b994d46a1e1fe216eb356ce P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:59.163420 1028 raft_consensus.cc:374] T 8f5e6ff35b994d46a1e1fe216eb356ce P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:59.163619 1028 raft_consensus.cc:729] T 8f5e6ff35b994d46a1e1fe216eb356ce P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:12:59.163906 1015 ts_tablet_manager.cc:1398] T 8f5e6ff35b994d46a1e1fe216eb356ce P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:59.163928 1028 consensus_queue.cc:260] T 8f5e6ff35b994d46a1e1fe216eb356ce P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:59.164667 981 raft_consensus.cc:504] T 33734f77af5342af82ee295bcf71f85a P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:59.165601 981 leader_election.cc:288] T 33734f77af5342af82ee295bcf71f85a P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Requested vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:12:59.165892 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33734f77af5342af82ee295bcf71f85a" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:12:59.166067 437 raft_consensus.cc:3037] T 33734f77af5342af82ee295bcf71f85a P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:59.166821 1028 ts_tablet_manager.cc:1398] T 8f5e6ff35b994d46a1e1fe216eb356ce P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:12:59.168690 437 raft_consensus.cc:2454] T 33734f77af5342af82ee295bcf71f85a P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:12:59.169232 29940 leader_election.cc:302] T 33734f77af5342af82ee295bcf71f85a P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:12:59.169643 981 raft_consensus.cc:2781] T 33734f77af5342af82ee295bcf71f85a P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:59.169885 981 raft_consensus.cc:686] T 33734f77af5342af82ee295bcf71f85a P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Becoming Leader. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Running, Role: LEADER
I20240503 02:12:59.169930 980 ts_tablet_manager.cc:1888] T 445caf2423c1488b8e47474032e68488 P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:59.170133 980 log.cc:1196] T 445caf2423c1488b8e47474032e68488 P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/445caf2423c1488b8e47474032e68488
I20240503 02:12:59.170254 981 consensus_queue.cc:237] T 33734f77af5342af82ee295bcf71f85a P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:59.170728 980 ts_tablet_manager.cc:1909] T 445caf2423c1488b8e47474032e68488 P 16440e365430405fb10366ab70e2b429: Deleting consensus metadata
I20240503 02:12:59.171940 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet 445caf2423c1488b8e47474032e68488 (table rebalance_test_table_12 [id=c501c7cdd0d142858bc57dffeae345f8]) successfully deleted
I20240503 02:12:59.172001 657 ts_tablet_manager.cc:1888] T 445caf2423c1488b8e47474032e68488 P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:59.172122 657 log.cc:1196] T 445caf2423c1488b8e47474032e68488 P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/445caf2423c1488b8e47474032e68488
I20240503 02:12:59.172461 980 tablet_replica.cc:330] T a74d38f7d6e5441396944c52acd41c74 P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:12:59.172552 657 ts_tablet_manager.cc:1909] T 445caf2423c1488b8e47474032e68488 P 155735c7f8b14997ac4c770aee2a9401: Deleting consensus metadata
I20240503 02:12:59.172806 980 raft_consensus.cc:2229] T a74d38f7d6e5441396944c52acd41c74 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:59.173987 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33734f77af5342af82ee295bcf71f85a" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:12:59.174259 567 raft_consensus.cc:2454] T 33734f77af5342af82ee295bcf71f85a P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:12:59.174649 568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33734f77af5342af82ee295bcf71f85a" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:12:59.174819 568 raft_consensus.cc:3037] T 33734f77af5342af82ee295bcf71f85a P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:59.174958 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet 445caf2423c1488b8e47474032e68488 (table rebalance_test_table_12 [id=c501c7cdd0d142858bc57dffeae345f8]) successfully deleted
I20240503 02:12:59.175827 1017 tablet_bootstrap.cc:492] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:12:59.176004 1017 ts_tablet_manager.cc:1367] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.042s user 0.001s sys 0.013s
I20240503 02:12:59.176836 1017 raft_consensus.cc:348] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:12:59.177726 29854 tablet_service.cc:1514] Processing DeleteTablet for tablet a592cdf6d1d747c68b5334b90006fa8b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:59 UTC) from {username='slave'} at 127.0.0.1:58744
I20240503 02:12:59.177716 568 raft_consensus.cc:2454] T 33734f77af5342af82ee295bcf71f85a P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:12:59.178478 1017 raft_consensus.cc:374] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:59.178576 1017 raft_consensus.cc:729] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:12:59.178920 1017 consensus_queue.cc:260] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:12:59.180425 1017 ts_tablet_manager.cc:1398] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:12:59.182636 1017 tablet_bootstrap.cc:492] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:12:59.183765 980 raft_consensus.cc:2258] T a74d38f7d6e5441396944c52acd41c74 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:59.184770 980 ts_tablet_manager.cc:1875] T a74d38f7d6e5441396944c52acd41c74 P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:59.184646 1017 tablet_bootstrap.cc:654] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:59.185238 1025 tablet_bootstrap.cc:492] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:59.185724 1025 ts_tablet_manager.cc:1367] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.044s user 0.001s sys 0.008s
I20240503 02:12:59.186539 1025 raft_consensus.cc:348] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:59.186761 1025 raft_consensus.cc:374] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:59.186877 1025 raft_consensus.cc:729] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:59.187144 1025 consensus_queue.cc:260] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:59.187758 1025 ts_tablet_manager.cc:1398] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:59.188033 1025 tablet_bootstrap.cc:492] T ba97049157a646c780fa203d34cd4c82 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:12:59.189528 702 raft_consensus.cc:482] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:59.189651 1025 tablet_bootstrap.cc:654] T ba97049157a646c780fa203d34cd4c82 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:59.189700 702 raft_consensus.cc:504] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:59.189993 1024 ts_tablet_manager.cc:1888] T a592cdf6d1d747c68b5334b90006fa8b P a9094240860e43bfb129193f3bfd5be6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:59.190120 1024 log.cc:1196] T a592cdf6d1d747c68b5334b90006fa8b P a9094240860e43bfb129193f3bfd5be6: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/wals/a592cdf6d1d747c68b5334b90006fa8b
I20240503 02:12:59.190606 1024 ts_tablet_manager.cc:1909] T a592cdf6d1d747c68b5334b90006fa8b P a9094240860e43bfb129193f3bfd5be6: Deleting consensus metadata
I20240503 02:12:59.190611 702 leader_election.cc:288] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:12:59.191471 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b8ebcf6f3245e6bd93eeb38ca1cbf5" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:12:59.191650 1024 tablet_replica.cc:330] T a74d38f7d6e5441396944c52acd41c74 P a9094240860e43bfb129193f3bfd5be6: stopping tablet replica
I20240503 02:12:59.191758 30004 raft_consensus.cc:2454] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:12:59.191879 1024 raft_consensus.cc:2229] T a74d38f7d6e5441396944c52acd41c74 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:59.192081 1024 raft_consensus.cc:2258] T a74d38f7d6e5441396944c52acd41c74 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:59.192788 1024 ts_tablet_manager.cc:1875] T a74d38f7d6e5441396944c52acd41c74 P a9094240860e43bfb129193f3bfd5be6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:59.193624 29730 catalog_manager.cc:5484] T 33734f77af5342af82ee295bcf71f85a P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: term changed from 0 to 1, leader changed from <none> to 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2). New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:12:59.194295 29809 leader_election.cc:302] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429; no voters:
I20240503 02:12:59.198328 29718 catalog_manager.cc:4830] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): tablet a592cdf6d1d747c68b5334b90006fa8b (table rebalance_test_table_12 [id=c501c7cdd0d142858bc57dffeae345f8]) successfully deleted
I20240503 02:12:59.201927 568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b8ebcf6f3245e6bd93eeb38ca1cbf5" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:12:59.202221 702 raft_consensus.cc:2781] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:59.202364 702 raft_consensus.cc:482] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:59.202474 702 raft_consensus.cc:3037] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:59.203109 1017 tablet_bootstrap.cc:492] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:12:59.203338 1017 ts_tablet_manager.cc:1367] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.007s
I20240503 02:12:59.203554 980 ts_tablet_manager.cc:1888] T a74d38f7d6e5441396944c52acd41c74 P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:59.203763 980 log.cc:1196] T a74d38f7d6e5441396944c52acd41c74 P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/a74d38f7d6e5441396944c52acd41c74
I20240503 02:12:59.204357 980 ts_tablet_manager.cc:1909] T a74d38f7d6e5441396944c52acd41c74 P 16440e365430405fb10366ab70e2b429: Deleting consensus metadata
I20240503 02:12:59.204381 1017 raft_consensus.cc:348] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:59.204710 1017 raft_consensus.cc:374] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:59.204869 1017 raft_consensus.cc:729] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
W20240503 02:12:59.209393 29808 leader_election.cc:341] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:59.209276 1017 consensus_queue.cc:260] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:59.209965 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet a74d38f7d6e5441396944c52acd41c74 (table rebalance_test_table_12 [id=c501c7cdd0d142858bc57dffeae345f8]) successfully deleted
W20240503 02:12:59.210266 29721 catalog_manager.cc:4797] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): delete failed for tablet a592cdf6d1d747c68b5334b90006fa8b because the tablet was not found. No further retry: Not found: Tablet not found: a592cdf6d1d747c68b5334b90006fa8b
I20240503 02:12:59.210530 1017 ts_tablet_manager.cc:1398] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240503 02:12:59.212608 1025 tablet_bootstrap.cc:492] T ba97049157a646c780fa203d34cd4c82 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:12:59.212859 1025 ts_tablet_manager.cc:1367] T ba97049157a646c780fa203d34cd4c82 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.001s
I20240503 02:12:59.213493 656 ts_tablet_manager.cc:1888] T a74d38f7d6e5441396944c52acd41c74 P 1f10f80f13c84e80b03eda7085a3b012: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:59.213672 656 log.cc:1196] T a74d38f7d6e5441396944c52acd41c74 P 1f10f80f13c84e80b03eda7085a3b012: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/wals/a74d38f7d6e5441396944c52acd41c74
I20240503 02:12:59.213909 1025 raft_consensus.cc:348] T ba97049157a646c780fa203d34cd4c82 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:12:59.214179 1025 raft_consensus.cc:374] T ba97049157a646c780fa203d34cd4c82 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:59.214262 1025 raft_consensus.cc:729] T ba97049157a646c780fa203d34cd4c82 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:12:59.216163 1025 consensus_queue.cc:260] T ba97049157a646c780fa203d34cd4c82 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:12:59.216418 656 ts_tablet_manager.cc:1909] T a74d38f7d6e5441396944c52acd41c74 P 1f10f80f13c84e80b03eda7085a3b012: Deleting consensus metadata
I20240503 02:12:59.216810 1025 ts_tablet_manager.cc:1398] T ba97049157a646c780fa203d34cd4c82 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:12:59.217643 29719 catalog_manager.cc:4830] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): tablet a74d38f7d6e5441396944c52acd41c74 (table rebalance_test_table_12 [id=c501c7cdd0d142858bc57dffeae345f8]) successfully deleted
I20240503 02:12:59.205706 702 raft_consensus.cc:504] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:59.219442 702 leader_election.cc:288] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Requested vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:12:59.219941 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b8ebcf6f3245e6bd93eeb38ca1cbf5" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:12:59.220186 30004 raft_consensus.cc:3037] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:59.220533 1024 ts_tablet_manager.cc:1888] T a74d38f7d6e5441396944c52acd41c74 P a9094240860e43bfb129193f3bfd5be6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:59.220670 1024 log.cc:1196] T a74d38f7d6e5441396944c52acd41c74 P a9094240860e43bfb129193f3bfd5be6: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/wals/a74d38f7d6e5441396944c52acd41c74
I20240503 02:12:59.221956 568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b8ebcf6f3245e6bd93eeb38ca1cbf5" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:12:59.222154 568 raft_consensus.cc:3037] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:59.222313 1024 ts_tablet_manager.cc:1909] T a74d38f7d6e5441396944c52acd41c74 P a9094240860e43bfb129193f3bfd5be6: Deleting consensus metadata
I20240503 02:12:59.222961 30004 raft_consensus.cc:2454] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:12:59.223779 29809 leader_election.cc:302] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429; no voters:
I20240503 02:12:59.223851 29718 catalog_manager.cc:4830] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): tablet a74d38f7d6e5441396944c52acd41c74 (table rebalance_test_table_12 [id=c501c7cdd0d142858bc57dffeae345f8]) successfully deleted
I20240503 02:12:59.224188 702 raft_consensus.cc:2781] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:59.224722 568 raft_consensus.cc:2454] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:12:59.225543 702 raft_consensus.cc:686] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Becoming Leader. State: Replica: 16440e365430405fb10366ab70e2b429, State: Running, Role: LEADER
I20240503 02:12:59.225899 702 consensus_queue.cc:237] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:12:59.230216 29729 catalog_manager.cc:5484] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 16440e365430405fb10366ab70e2b429 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16440e365430405fb10366ab70e2b429 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:12:59.238322 877 raft_consensus.cc:482] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:59.238498 877 raft_consensus.cc:504] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:12:59.239233 877 leader_election.cc:288] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:12:59.241683 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba97049157a646c780fa203d34cd4c82" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:12:59.242018 437 raft_consensus.cc:2454] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 0.
I20240503 02:12:59.242843 502 leader_election.cc:302] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f575ce92c0224eeb80b9fae5f6561fbe, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:12:59.243189 876 raft_consensus.cc:2781] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:59.243332 876 raft_consensus.cc:482] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:59.243445 876 raft_consensus.cc:3037] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:59.247222 876 raft_consensus.cc:504] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:12:59.247687 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba97049157a646c780fa203d34cd4c82" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:12:59.247926 30266 raft_consensus.cc:2454] T ba97049157a646c780fa203d34cd4c82 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 0.
I20240503 02:12:59.248327 876 leader_election.cc:288] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 election: Requested vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:12:59.248404 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba97049157a646c780fa203d34cd4c82" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:12:59.248641 30264 raft_consensus.cc:3037] T ba97049157a646c780fa203d34cd4c82 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:59.249534 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba97049157a646c780fa203d34cd4c82" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:12:59.249775 437 raft_consensus.cc:3037] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:59.251192 30264 raft_consensus.cc:2454] T ba97049157a646c780fa203d34cd4c82 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 1.
I20240503 02:12:59.251590 502 leader_election.cc:302] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:12:59.252727 437 raft_consensus.cc:2454] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 1.
I20240503 02:12:59.253005 876 raft_consensus.cc:2781] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:59.253568 876 raft_consensus.cc:686] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 LEADER]: Becoming Leader. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Running, Role: LEADER
I20240503 02:12:59.253858 876 consensus_queue.cc:237] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:12:59.257393 29729 catalog_manager.cc:5484] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: term changed from 0 to 1, leader changed from <none> to f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8). New cstate: current_term: 1 leader_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:12:59.257869 30004 consensus_queue.cc:237] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:59.260524 30134 raft_consensus.cc:1261] T 35a5a7f4fe184b9fadc94c98b61deaf0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:12:59.260684 29940 consensus_peers.cc:485] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): Couldn't send request to peer fbbc2be6382e47908ecb7c68c367bd26. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 35a5a7f4fe184b9fadc94c98b61deaf0. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:59.260879 29873 raft_consensus.cc:1261] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:12:59.261437 981 consensus_queue.cc:1035] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:12:59.261916 981 consensus_queue.cc:1035] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:12:59.266048 981 raft_consensus.cc:2932] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:12:59.266726 30134 raft_consensus.cc:2932] T 35a5a7f4fe184b9fadc94c98b61deaf0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:12:59.270418 29873 raft_consensus.cc:2932] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:12:59.272877 29729 catalog_manager.cc:5484] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: config changed from index -1 to 4, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:12:59.310263 29730 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:12:59.317561 437 raft_consensus.cc:1261] T 33734f77af5342af82ee295bcf71f85a P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:59.317632 568 raft_consensus.cc:1261] T 33734f77af5342af82ee295bcf71f85a P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:59.318202 981 consensus_queue.cc:1035] T 33734f77af5342af82ee295bcf71f85a P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:59.318800 981 consensus_queue.cc:1035] T 33734f77af5342af82ee295bcf71f85a P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:59.322764 29725 catalog_manager.cc:5576] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233) does not have the latest schema for tablet 33734f77af5342af82ee295bcf71f85a (table rebalancer_dynamic_table [id=4c683f284ec140f69703e915a553a141]). Expected version 1 got 0
I20240503 02:12:59.323868 1069 tablet.cc:1722] T 33734f77af5342af82ee295bcf71f85a P 155735c7f8b14997ac4c770aee2a9401: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:59.324548 1069 tablet.cc:1620] T 33734f77af5342af82ee295bcf71f85a P 155735c7f8b14997ac4c770aee2a9401: MemRowSet was empty: no flush needed.
I20240503 02:12:59.334434 1070 tablet.cc:1722] T 33734f77af5342af82ee295bcf71f85a P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:59.335796 1070 tablet.cc:1620] T 33734f77af5342af82ee295bcf71f85a P f575ce92c0224eeb80b9fae5f6561fbe: MemRowSet was empty: no flush needed.
I20240503 02:12:59.339403 29730 catalog_manager.cc:5576] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233) does not have the latest schema for tablet 33734f77af5342af82ee295bcf71f85a (table rebalancer_dynamic_table [id=4c683f284ec140f69703e915a553a141]). Expected version 1 got 0
I20240503 02:12:59.341977 29725 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=4c683f284ec140f69703e915a553a141] alter complete (version 1)
I20240503 02:12:59.349864 29730 catalog_manager.cc:5576] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241) does not have the latest schema for tablet 33734f77af5342af82ee295bcf71f85a (table rebalancer_dynamic_table [id=4c683f284ec140f69703e915a553a141]). Expected version 1 got 0
I20240503 02:12:59.360579 1073 tablet.cc:1722] T 33734f77af5342af82ee295bcf71f85a P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:59.362174 1073 tablet.cc:1620] T 33734f77af5342af82ee295bcf71f85a P fbbc2be6382e47908ecb7c68c367bd26: MemRowSet was empty: no flush needed.
I20240503 02:12:59.363287 29725 catalog_manager.cc:5576] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241) does not have the latest schema for tablet 33734f77af5342af82ee295bcf71f85a (table rebalancer_dynamic_table [id=4c683f284ec140f69703e915a553a141]). Expected version 1 got 0
I20240503 02:12:59.399960 32467 raft_consensus.cc:482] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:59.400148 32467 raft_consensus.cc:504] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:59.401197 32467 leader_election.cc:288] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:12:59.411553 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f5e6ff35b994d46a1e1fe216eb356ce" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:12:59.411796 307 raft_consensus.cc:2454] T 8f5e6ff35b994d46a1e1fe216eb356ce P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:59.412297 30200 leader_election.cc:302] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, 8a666e798fe8477d8dd1e51b385d5ead; no voters:
I20240503 02:12:59.412595 913 raft_consensus.cc:2781] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:59.412691 913 raft_consensus.cc:482] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:59.412794 913 raft_consensus.cc:3037] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:59.415971 913 raft_consensus.cc:504] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:59.416702 913 leader_election.cc:288] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Requested vote from peers 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:12:59.417553 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f5e6ff35b994d46a1e1fe216eb356ce" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:12:59.417739 307 raft_consensus.cc:3037] T 8f5e6ff35b994d46a1e1fe216eb356ce P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:59.418414 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f5e6ff35b994d46a1e1fe216eb356ce" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:12:59.418712 436 raft_consensus.cc:2454] T 8f5e6ff35b994d46a1e1fe216eb356ce P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:12:59.419633 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f5e6ff35b994d46a1e1fe216eb356ce" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:12:59.419883 436 raft_consensus.cc:3037] T 8f5e6ff35b994d46a1e1fe216eb356ce P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:59.420527 307 raft_consensus.cc:2454] T 8f5e6ff35b994d46a1e1fe216eb356ce P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:59.423028 30200 leader_election.cc:302] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, 8a666e798fe8477d8dd1e51b385d5ead; no voters:
I20240503 02:12:59.423075 436 raft_consensus.cc:2454] T 8f5e6ff35b994d46a1e1fe216eb356ce P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:12:59.423449 913 raft_consensus.cc:2781] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:59.423838 913 raft_consensus.cc:686] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Becoming Leader. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Running, Role: LEADER
I20240503 02:12:59.424142 913 consensus_queue.cc:237] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:12:59.445324 29725 catalog_manager.cc:5484] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4). New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:12:59.450841 29725 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:12:59.460279 1069 tablet.cc:1722] T 33734f77af5342af82ee295bcf71f85a P 155735c7f8b14997ac4c770aee2a9401: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:12:59.460798 1069 tablet.cc:1620] T 33734f77af5342af82ee295bcf71f85a P 155735c7f8b14997ac4c770aee2a9401: MemRowSet was empty: no flush needed.
I20240503 02:12:59.461827 1070 tablet.cc:1722] T 33734f77af5342af82ee295bcf71f85a P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:12:59.462275 1070 tablet.cc:1620] T 33734f77af5342af82ee295bcf71f85a P f575ce92c0224eeb80b9fae5f6561fbe: MemRowSet was empty: no flush needed.
I20240503 02:12:59.472035 1073 tablet.cc:1722] T 33734f77af5342af82ee295bcf71f85a P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:12:59.472529 1073 tablet.cc:1620] T 33734f77af5342af82ee295bcf71f85a P fbbc2be6382e47908ecb7c68c367bd26: MemRowSet was empty: no flush needed.
I20240503 02:12:59.472796 29734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=4c683f284ec140f69703e915a553a141] alter complete (version 2)
I20240503 02:12:59.584993 29729 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:12:59.597412 1069 tablet.cc:1722] T 33734f77af5342af82ee295bcf71f85a P 155735c7f8b14997ac4c770aee2a9401: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:12:59.601866 1070 tablet.cc:1722] T 33734f77af5342af82ee295bcf71f85a P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:12:59.601963 1073 tablet.cc:1722] T 33734f77af5342af82ee295bcf71f85a P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:12:59.616060 29729 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table_new_name [id=4c683f284ec140f69703e915a553a141] alter complete (version 3)
I20240503 02:12:59.630200 29725 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240503 02:12:59.630872 29725 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240503 02:12:59.638406 29725 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet f9dc35fb35834c5d9cb5cea546eccbf8
I20240503 02:12:59.639278 30244 tablet_service.cc:1514] Processing DeleteTablet for tablet f9dc35fb35834c5d9cb5cea546eccbf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:59 UTC) from {username='slave'} at 127.0.0.1:49856
I20240503 02:12:59.639582 656 tablet_replica.cc:330] T f9dc35fb35834c5d9cb5cea546eccbf8 P 1f10f80f13c84e80b03eda7085a3b012: stopping tablet replica
I20240503 02:12:59.639801 656 raft_consensus.cc:2229] T f9dc35fb35834c5d9cb5cea546eccbf8 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:59.639966 29725 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet db3887ced8964b8196a78a55a1190af4
I20240503 02:12:59.640095 656 raft_consensus.cc:2258] T f9dc35fb35834c5d9cb5cea546eccbf8 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:59.640487 29854 tablet_service.cc:1514] Processing DeleteTablet for tablet f9dc35fb35834c5d9cb5cea546eccbf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:59 UTC) from {username='slave'} at 127.0.0.1:58744
I20240503 02:12:59.640861 656 ts_tablet_manager.cc:1875] T f9dc35fb35834c5d9cb5cea546eccbf8 P 1f10f80f13c84e80b03eda7085a3b012: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:59.641424 980 tablet_replica.cc:330] T f9dc35fb35834c5d9cb5cea546eccbf8 P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:12:59.641690 980 raft_consensus.cc:2229] T f9dc35fb35834c5d9cb5cea546eccbf8 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:59.641963 980 raft_consensus.cc:2258] T f9dc35fb35834c5d9cb5cea546eccbf8 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:59.642858 29984 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:12:59.642863 980 ts_tablet_manager.cc:1875] T f9dc35fb35834c5d9cb5cea546eccbf8 P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:59.643393 29986 tablet_service.cc:1514] Processing DeleteTablet for tablet f9dc35fb35834c5d9cb5cea546eccbf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:59 UTC) from {username='slave'} at 127.0.0.1:52158
I20240503 02:12:59.644006 29853 tablet_service.cc:1514] Processing DeleteTablet for tablet db3887ced8964b8196a78a55a1190af4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:59 UTC) from {username='slave'} at 127.0.0.1:58744
I20240503 02:12:59.644263 657 tablet_replica.cc:330] T f9dc35fb35834c5d9cb5cea546eccbf8 P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:12:59.644485 657 raft_consensus.cc:2229] T f9dc35fb35834c5d9cb5cea546eccbf8 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:59.644824 657 raft_consensus.cc:2258] T f9dc35fb35834c5d9cb5cea546eccbf8 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:59.645565 657 ts_tablet_manager.cc:1875] T f9dc35fb35834c5d9cb5cea546eccbf8 P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:59.651535 980 ts_tablet_manager.cc:1888] T f9dc35fb35834c5d9cb5cea546eccbf8 P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:59.651676 980 log.cc:1196] T f9dc35fb35834c5d9cb5cea546eccbf8 P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/f9dc35fb35834c5d9cb5cea546eccbf8
I20240503 02:12:59.652134 980 ts_tablet_manager.cc:1909] T f9dc35fb35834c5d9cb5cea546eccbf8 P 16440e365430405fb10366ab70e2b429: Deleting consensus metadata
I20240503 02:12:59.653641 30244 tablet_service.cc:1514] Processing DeleteTablet for tablet db3887ced8964b8196a78a55a1190af4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:59 UTC) from {username='slave'} at 127.0.0.1:49856
I20240503 02:12:59.654218 29725 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet ae187b76f8834ba0a7a8b1a3bf33e8e4
I20240503 02:12:59.654250 30114 tablet_service.cc:1514] Processing DeleteTablet for tablet db3887ced8964b8196a78a55a1190af4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:59 UTC) from {username='slave'} at 127.0.0.1:44812
I20240503 02:12:59.654548 1024 tablet_replica.cc:330] T db3887ced8964b8196a78a55a1190af4 P a9094240860e43bfb129193f3bfd5be6: stopping tablet replica
I20240503 02:12:59.654750 1024 raft_consensus.cc:2229] T db3887ced8964b8196a78a55a1190af4 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:59.655011 1024 raft_consensus.cc:2258] T db3887ced8964b8196a78a55a1190af4 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:59.655131 30244 tablet_service.cc:1514] Processing DeleteTablet for tablet ae187b76f8834ba0a7a8b1a3bf33e8e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:59 UTC) from {username='slave'} at 127.0.0.1:49856
I20240503 02:12:59.655642 29986 tablet_service.cc:1514] Processing DeleteTablet for tablet ae187b76f8834ba0a7a8b1a3bf33e8e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:59 UTC) from {username='slave'} at 127.0.0.1:52158
I20240503 02:12:59.655695 1024 ts_tablet_manager.cc:1875] T db3887ced8964b8196a78a55a1190af4 P a9094240860e43bfb129193f3bfd5be6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:59.656816 980 tablet_replica.cc:330] T db3887ced8964b8196a78a55a1190af4 P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:12:59.657084 980 raft_consensus.cc:2229] T db3887ced8964b8196a78a55a1190af4 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:59.657289 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet f9dc35fb35834c5d9cb5cea546eccbf8 (table rebalance_test_table_14 [id=079f836c26424ee4adc122852f327cc7]) successfully deleted
I20240503 02:12:59.657343 980 raft_consensus.cc:2258] T db3887ced8964b8196a78a55a1190af4 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:59.658179 980 ts_tablet_manager.cc:1875] T db3887ced8964b8196a78a55a1190af4 P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:59.658336 29853 tablet_service.cc:1514] Processing DeleteTablet for tablet ae187b76f8834ba0a7a8b1a3bf33e8e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:59 UTC) from {username='slave'} at 127.0.0.1:58744
I20240503 02:12:59.667191 1024 ts_tablet_manager.cc:1888] T db3887ced8964b8196a78a55a1190af4 P a9094240860e43bfb129193f3bfd5be6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:59.672181 656 ts_tablet_manager.cc:1888] T f9dc35fb35834c5d9cb5cea546eccbf8 P 1f10f80f13c84e80b03eda7085a3b012: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:59.672495 1024 log.cc:1196] T db3887ced8964b8196a78a55a1190af4 P a9094240860e43bfb129193f3bfd5be6: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/wals/db3887ced8964b8196a78a55a1190af4
I20240503 02:12:59.673200 1024 ts_tablet_manager.cc:1909] T db3887ced8964b8196a78a55a1190af4 P a9094240860e43bfb129193f3bfd5be6: Deleting consensus metadata
I20240503 02:12:59.673393 656 log.cc:1196] T f9dc35fb35834c5d9cb5cea546eccbf8 P 1f10f80f13c84e80b03eda7085a3b012: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/wals/f9dc35fb35834c5d9cb5cea546eccbf8
I20240503 02:12:59.674012 656 ts_tablet_manager.cc:1909] T f9dc35fb35834c5d9cb5cea546eccbf8 P 1f10f80f13c84e80b03eda7085a3b012: Deleting consensus metadata
I20240503 02:12:59.674202 980 ts_tablet_manager.cc:1888] T db3887ced8964b8196a78a55a1190af4 P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:59.674361 980 log.cc:1196] T db3887ced8964b8196a78a55a1190af4 P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/db3887ced8964b8196a78a55a1190af4
I20240503 02:12:59.674824 980 ts_tablet_manager.cc:1909] T db3887ced8964b8196a78a55a1190af4 P 16440e365430405fb10366ab70e2b429: Deleting consensus metadata
I20240503 02:12:59.676080 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet db3887ced8964b8196a78a55a1190af4 (table rebalance_test_table_14 [id=079f836c26424ee4adc122852f327cc7]) successfully deleted
I20240503 02:12:59.677067 29718 catalog_manager.cc:4830] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): tablet db3887ced8964b8196a78a55a1190af4 (table rebalance_test_table_14 [id=079f836c26424ee4adc122852f327cc7]) successfully deleted
I20240503 02:12:59.677294 29719 catalog_manager.cc:4830] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): tablet f9dc35fb35834c5d9cb5cea546eccbf8 (table rebalance_test_table_14 [id=079f836c26424ee4adc122852f327cc7]) successfully deleted
I20240503 02:12:59.677902 656 tablet_replica.cc:330] T db3887ced8964b8196a78a55a1190af4 P 1f10f80f13c84e80b03eda7085a3b012: stopping tablet replica
I20240503 02:12:59.678241 656 raft_consensus.cc:2229] T db3887ced8964b8196a78a55a1190af4 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:59.678700 656 raft_consensus.cc:2258] T db3887ced8964b8196a78a55a1190af4 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:59.679859 656 ts_tablet_manager.cc:1875] T db3887ced8964b8196a78a55a1190af4 P 1f10f80f13c84e80b03eda7085a3b012: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:59.681653 980 tablet_replica.cc:330] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:12:59.681944 980 raft_consensus.cc:2229] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:59.683010 980 raft_consensus.cc:2258] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:59.683935 980 ts_tablet_manager.cc:1875] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:59.692610 657 ts_tablet_manager.cc:1888] T f9dc35fb35834c5d9cb5cea546eccbf8 P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:59.692814 657 log.cc:1196] T f9dc35fb35834c5d9cb5cea546eccbf8 P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/f9dc35fb35834c5d9cb5cea546eccbf8
I20240503 02:12:59.693149 656 ts_tablet_manager.cc:1888] T db3887ced8964b8196a78a55a1190af4 P 1f10f80f13c84e80b03eda7085a3b012: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:59.693287 656 log.cc:1196] T db3887ced8964b8196a78a55a1190af4 P 1f10f80f13c84e80b03eda7085a3b012: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/wals/db3887ced8964b8196a78a55a1190af4
I20240503 02:12:59.694475 657 ts_tablet_manager.cc:1909] T f9dc35fb35834c5d9cb5cea546eccbf8 P 155735c7f8b14997ac4c770aee2a9401: Deleting consensus metadata
I20240503 02:12:59.695823 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet f9dc35fb35834c5d9cb5cea546eccbf8 (table rebalance_test_table_14 [id=079f836c26424ee4adc122852f327cc7]) successfully deleted
I20240503 02:12:59.695897 876 consensus_queue.cc:1035] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:59.696306 657 tablet_replica.cc:330] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:12:59.697592 657 raft_consensus.cc:2229] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:59.697932 657 raft_consensus.cc:2258] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:59.698689 657 ts_tablet_manager.cc:1875] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:59.701128 980 ts_tablet_manager.cc:1888] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:59.701301 980 log.cc:1196] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/ae187b76f8834ba0a7a8b1a3bf33e8e4
I20240503 02:12:59.701669 1042 consensus_queue.cc:1035] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:59.702172 980 ts_tablet_manager.cc:1909] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 16440e365430405fb10366ab70e2b429: Deleting consensus metadata
I20240503 02:12:59.703362 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet ae187b76f8834ba0a7a8b1a3bf33e8e4 (table rebalance_test_table_14 [id=079f836c26424ee4adc122852f327cc7]) successfully deleted
I20240503 02:12:59.707366 657 ts_tablet_manager.cc:1888] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:59.707669 657 log.cc:1196] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/ae187b76f8834ba0a7a8b1a3bf33e8e4
I20240503 02:12:59.708317 657 ts_tablet_manager.cc:1909] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 155735c7f8b14997ac4c770aee2a9401: Deleting consensus metadata
I20240503 02:12:59.710968 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet ae187b76f8834ba0a7a8b1a3bf33e8e4 (table rebalance_test_table_14 [id=079f836c26424ee4adc122852f327cc7]) successfully deleted
I20240503 02:12:59.722424 29725 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:12:59.722704 29725 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:12:59.727105 29725 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet 33734f77af5342af82ee295bcf71f85a
I20240503 02:12:59.728281 546 tablet_service.cc:1514] Processing DeleteTablet for tablet 33734f77af5342af82ee295bcf71f85a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:59 UTC) from {username='slave'} at 127.0.0.1:39028
I20240503 02:12:59.728546 29986 tablet_service.cc:1514] Processing DeleteTablet for tablet 33734f77af5342af82ee295bcf71f85a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:59 UTC) from {username='slave'} at 127.0.0.1:52158
I20240503 02:12:59.728848 657 tablet_replica.cc:330] T 33734f77af5342af82ee295bcf71f85a P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:12:59.729010 417 tablet_service.cc:1514] Processing DeleteTablet for tablet 33734f77af5342af82ee295bcf71f85a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:59 UTC) from {username='slave'} at 127.0.0.1:50814
I20240503 02:12:59.729305 657 raft_consensus.cc:2229] T 33734f77af5342af82ee295bcf71f85a P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:59.730237 657 raft_consensus.cc:2258] T 33734f77af5342af82ee295bcf71f85a P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:59.737430 1111 tablet_replica.cc:330] T 33734f77af5342af82ee295bcf71f85a P fbbc2be6382e47908ecb7c68c367bd26: stopping tablet replica
I20240503 02:12:59.737807 1111 raft_consensus.cc:2229] T 33734f77af5342af82ee295bcf71f85a P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:59.738143 1111 raft_consensus.cc:2258] T 33734f77af5342af82ee295bcf71f85a P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:59.739423 1111 ts_tablet_manager.cc:1875] T 33734f77af5342af82ee295bcf71f85a P fbbc2be6382e47908ecb7c68c367bd26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:59.746012 656 ts_tablet_manager.cc:1909] T db3887ced8964b8196a78a55a1190af4 P 1f10f80f13c84e80b03eda7085a3b012: Deleting consensus metadata
I20240503 02:12:59.747342 29719 catalog_manager.cc:4830] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): tablet db3887ced8964b8196a78a55a1190af4 (table rebalance_test_table_14 [id=079f836c26424ee4adc122852f327cc7]) successfully deleted
I20240503 02:12:59.748065 656 tablet_replica.cc:330] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 1f10f80f13c84e80b03eda7085a3b012: stopping tablet replica
I20240503 02:12:59.748272 656 raft_consensus.cc:2229] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:59.748618 656 raft_consensus.cc:2258] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:59.749365 656 ts_tablet_manager.cc:1875] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 1f10f80f13c84e80b03eda7085a3b012: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:59.751762 657 ts_tablet_manager.cc:1875] T 33734f77af5342af82ee295bcf71f85a P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:59.758524 29853 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:12:59.762588 1111 ts_tablet_manager.cc:1888] T 33734f77af5342af82ee295bcf71f85a P fbbc2be6382e47908ecb7c68c367bd26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:59.762781 1110 tablet_replica.cc:330] T 33734f77af5342af82ee295bcf71f85a P f575ce92c0224eeb80b9fae5f6561fbe: stopping tablet replica
I20240503 02:12:59.762861 1111 log.cc:1196] T 33734f77af5342af82ee295bcf71f85a P fbbc2be6382e47908ecb7c68c367bd26: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal/wals/33734f77af5342af82ee295bcf71f85a
I20240503 02:12:59.763058 1110 raft_consensus.cc:2229] T 33734f77af5342af82ee295bcf71f85a P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:59.763296 1110 raft_consensus.cc:2258] T 33734f77af5342af82ee295bcf71f85a P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:59.763638 1111 ts_tablet_manager.cc:1909] T 33734f77af5342af82ee295bcf71f85a P fbbc2be6382e47908ecb7c68c367bd26: Deleting consensus metadata
I20240503 02:12:59.764299 1110 ts_tablet_manager.cc:1875] T 33734f77af5342af82ee295bcf71f85a P f575ce92c0224eeb80b9fae5f6561fbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:59.764705 29719 catalog_manager.cc:4830] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): tablet 33734f77af5342af82ee295bcf71f85a (table rebalancer_dynamic_table_new_name [id=4c683f284ec140f69703e915a553a141]) successfully deleted
I20240503 02:12:59.770054 1112 ts_tablet_manager.cc:902] T 35a5a7f4fe184b9fadc94c98b61deaf0 P fbbc2be6382e47908ecb7c68c367bd26: Initiating tablet copy from peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:12:59.773108 876 consensus_queue.cc:1035] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:59.786504 657 ts_tablet_manager.cc:1888] T 33734f77af5342af82ee295bcf71f85a P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:59.786687 657 log.cc:1196] T 33734f77af5342af82ee295bcf71f85a P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/33734f77af5342af82ee295bcf71f85a
I20240503 02:12:59.787178 657 ts_tablet_manager.cc:1909] T 33734f77af5342af82ee295bcf71f85a P 155735c7f8b14997ac4c770aee2a9401: Deleting consensus metadata
I20240503 02:12:59.788336 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet 33734f77af5342af82ee295bcf71f85a (table rebalancer_dynamic_table_new_name [id=4c683f284ec140f69703e915a553a141]) successfully deleted
I20240503 02:12:59.789980 656 ts_tablet_manager.cc:1888] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 1f10f80f13c84e80b03eda7085a3b012: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:59.790132 656 log.cc:1196] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 1f10f80f13c84e80b03eda7085a3b012: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/wals/ae187b76f8834ba0a7a8b1a3bf33e8e4
I20240503 02:12:59.790617 656 ts_tablet_manager.cc:1909] T ae187b76f8834ba0a7a8b1a3bf33e8e4 P 1f10f80f13c84e80b03eda7085a3b012: Deleting consensus metadata
I20240503 02:12:59.791615 29719 catalog_manager.cc:4830] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): tablet ae187b76f8834ba0a7a8b1a3bf33e8e4 (table rebalance_test_table_14 [id=079f836c26424ee4adc122852f327cc7]) successfully deleted
I20240503 02:12:59.795552 1110 ts_tablet_manager.cc:1888] T 33734f77af5342af82ee295bcf71f85a P f575ce92c0224eeb80b9fae5f6561fbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:59.797328 1110 log.cc:1196] T 33734f77af5342af82ee295bcf71f85a P f575ce92c0224eeb80b9fae5f6561fbe: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal/wals/33734f77af5342af82ee295bcf71f85a
I20240503 02:12:59.798015 1110 ts_tablet_manager.cc:1909] T 33734f77af5342af82ee295bcf71f85a P f575ce92c0224eeb80b9fae5f6561fbe: Deleting consensus metadata
I20240503 02:12:59.799571 29718 catalog_manager.cc:4830] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): tablet 33734f77af5342af82ee295bcf71f85a (table rebalancer_dynamic_table_new_name [id=4c683f284ec140f69703e915a553a141]) successfully deleted
I20240503 02:12:59.804210 1112 tablet_copy_client.cc:308] T 35a5a7f4fe184b9fadc94c98b61deaf0 P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.2:37873
I20240503 02:12:59.807667 30026 tablet_copy_service.cc:140] P 155735c7f8b14997ac4c770aee2a9401: Received BeginTabletCopySession request for tablet 35a5a7f4fe184b9fadc94c98b61deaf0 from peer fbbc2be6382e47908ecb7c68c367bd26 ({username='slave'} at 127.22.108.7:60369)
I20240503 02:12:59.807873 30026 tablet_copy_service.cc:161] P 155735c7f8b14997ac4c770aee2a9401: Beginning new tablet copy session on tablet 35a5a7f4fe184b9fadc94c98b61deaf0 from peer fbbc2be6382e47908ecb7c68c367bd26 at {username='slave'} at 127.22.108.7:60369: session id = fbbc2be6382e47908ecb7c68c367bd26-35a5a7f4fe184b9fadc94c98b61deaf0
I20240503 02:12:59.810612 30114 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:12:59.811509 30026 tablet_copy_source_session.cc:205] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:59.819521 702 consensus_queue.cc:1035] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:59.830673 1112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35a5a7f4fe184b9fadc94c98b61deaf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.831310 29725 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:12:59.841725 32620 tablet_service.cc:1467] Processing CreateTablet for tablet ab995d6a8aeb42218230cf4f7cda448b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=24bee9471db248d98a23cbedf87d8255]), partition=
I20240503 02:12:59.841953 29986 tablet_service.cc:1467] Processing CreateTablet for tablet ab995d6a8aeb42218230cf4f7cda448b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=24bee9471db248d98a23cbedf87d8255]), partition=
I20240503 02:12:59.842170 32620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab995d6a8aeb42218230cf4f7cda448b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.842286 29986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab995d6a8aeb42218230cf4f7cda448b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.842356 546 tablet_service.cc:1467] Processing CreateTablet for tablet ab995d6a8aeb42218230cf4f7cda448b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=24bee9471db248d98a23cbedf87d8255]), partition=
I20240503 02:12:59.842727 546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab995d6a8aeb42218230cf4f7cda448b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.857533 1130 tablet_bootstrap.cc:492] T ab995d6a8aeb42218230cf4f7cda448b P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:12:59.864894 1131 tablet_bootstrap.cc:492] T ab995d6a8aeb42218230cf4f7cda448b P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:12:59.865581 1130 tablet_bootstrap.cc:654] T ab995d6a8aeb42218230cf4f7cda448b P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:59.880671 547 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:59.886171 32753 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:59.886423 417 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:59.887362 1131 tablet_bootstrap.cc:654] T ab995d6a8aeb42218230cf4f7cda448b P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:59.940557 1112 tablet_copy_client.cc:783] T 35a5a7f4fe184b9fadc94c98b61deaf0 P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:59.941079 1112 tablet_copy_client.cc:647] T 35a5a7f4fe184b9fadc94c98b61deaf0 P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:59.953409 29725 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_extra_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:59.954898 1131 tablet_bootstrap.cc:492] T ab995d6a8aeb42218230cf4f7cda448b P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:12:59.955132 1131 ts_tablet_manager.cc:1367] T ab995d6a8aeb42218230cf4f7cda448b P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.090s user 0.012s sys 0.008s
I20240503 02:12:59.961187 1112 tablet_copy_client.cc:523] T 35a5a7f4fe184b9fadc94c98b61deaf0 P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:59.961369 1132 tablet_bootstrap.cc:492] T ab995d6a8aeb42218230cf4f7cda448b P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:12:59.963085 1132 tablet_bootstrap.cc:654] T ab995d6a8aeb42218230cf4f7cda448b P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:59.964874 29853 tablet_service.cc:1467] Processing CreateTablet for tablet f82e505ac13a41019e422b9dc1e4fef9 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=c14bbf6b6bfd41148c82b54717aceb15]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:59.965327 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f82e505ac13a41019e422b9dc1e4fef9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.965859 29986 tablet_service.cc:1467] Processing CreateTablet for tablet 9102ab9545c54e82b642673795cb747a (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=c14bbf6b6bfd41148c82b54717aceb15]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:59.965935 417 tablet_service.cc:1467] Processing CreateTablet for tablet f82e505ac13a41019e422b9dc1e4fef9 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=c14bbf6b6bfd41148c82b54717aceb15]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:59.966233 29986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9102ab9545c54e82b642673795cb747a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.966333 417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f82e505ac13a41019e422b9dc1e4fef9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.969417 416 tablet_service.cc:1467] Processing CreateTablet for tablet 9102ab9545c54e82b642673795cb747a (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=c14bbf6b6bfd41148c82b54717aceb15]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:59.969610 32754 tablet_service.cc:1467] Processing CreateTablet for tablet b7b284cffcb04ce883a24f6a8796748f (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=c14bbf6b6bfd41148c82b54717aceb15]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:59.969826 416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9102ab9545c54e82b642673795cb747a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.970067 32754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7b284cffcb04ce883a24f6a8796748f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.970099 29854 tablet_service.cc:1467] Processing CreateTablet for tablet 9102ab9545c54e82b642673795cb747a (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=c14bbf6b6bfd41148c82b54717aceb15]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:59.970487 29854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9102ab9545c54e82b642673795cb747a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.973054 32753 tablet_service.cc:1467] Processing CreateTablet for tablet f82e505ac13a41019e422b9dc1e4fef9 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=c14bbf6b6bfd41148c82b54717aceb15]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:59.973443 32753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f82e505ac13a41019e422b9dc1e4fef9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.976778 32620 tablet_service.cc:1467] Processing CreateTablet for tablet b7b284cffcb04ce883a24f6a8796748f (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=c14bbf6b6bfd41148c82b54717aceb15]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:59.977485 415 tablet_service.cc:1467] Processing CreateTablet for tablet b7b284cffcb04ce883a24f6a8796748f (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=c14bbf6b6bfd41148c82b54717aceb15]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:59.978006 415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7b284cffcb04ce883a24f6a8796748f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.978322 32620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7b284cffcb04ce883a24f6a8796748f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.984931 30244 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:12:59.990809 913 consensus_queue.cc:1035] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:59.995059 1136 tablet_bootstrap.cc:492] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:00.000149 1136 tablet_bootstrap.cc:654] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.002609 1112 tablet_bootstrap.cc:492] T 35a5a7f4fe184b9fadc94c98b61deaf0 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:00.027980 1137 tablet_bootstrap.cc:492] T b7b284cffcb04ce883a24f6a8796748f P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:00.030282 913 consensus_queue.cc:1035] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:00.030861 1137 tablet_bootstrap.cc:654] T b7b284cffcb04ce883a24f6a8796748f P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.072243 1130 tablet_bootstrap.cc:492] T ab995d6a8aeb42218230cf4f7cda448b P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:13:00.072515 1130 ts_tablet_manager.cc:1367] T ab995d6a8aeb42218230cf4f7cda448b P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.215s user 0.015s sys 0.008s
I20240503 02:13:00.073518 1130 raft_consensus.cc:348] T ab995d6a8aeb42218230cf4f7cda448b P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:00.075018 1130 raft_consensus.cc:374] T ab995d6a8aeb42218230cf4f7cda448b P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.075218 1130 raft_consensus.cc:729] T ab995d6a8aeb42218230cf4f7cda448b P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.075660 1130 consensus_queue.cc:260] T ab995d6a8aeb42218230cf4f7cda448b P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:00.088011 1130 ts_tablet_manager.cc:1398] T ab995d6a8aeb42218230cf4f7cda448b P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240503 02:13:00.089814 1130 tablet_bootstrap.cc:492] T 9102ab9545c54e82b642673795cb747a P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:13:00.091856 1130 tablet_bootstrap.cc:654] T 9102ab9545c54e82b642673795cb747a P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.094385 1130 tablet_bootstrap.cc:492] T 9102ab9545c54e82b642673795cb747a P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:13:00.094640 1130 ts_tablet_manager.cc:1367] T 9102ab9545c54e82b642673795cb747a P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:13:00.095510 1130 raft_consensus.cc:348] T 9102ab9545c54e82b642673795cb747a P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:00.095829 1130 raft_consensus.cc:374] T 9102ab9545c54e82b642673795cb747a P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.095988 1130 raft_consensus.cc:729] T 9102ab9545c54e82b642673795cb747a P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.096318 1130 consensus_queue.cc:260] T 9102ab9545c54e82b642673795cb747a P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:00.097118 1130 ts_tablet_manager.cc:1398] T 9102ab9545c54e82b642673795cb747a P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:00.102681 1135 tablet_bootstrap.cc:492] T f82e505ac13a41019e422b9dc1e4fef9 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:13:00.107057 1132 tablet_bootstrap.cc:492] T ab995d6a8aeb42218230cf4f7cda448b P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:00.107285 1132 ts_tablet_manager.cc:1367] T ab995d6a8aeb42218230cf4f7cda448b P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.146s user 0.003s sys 0.000s
I20240503 02:13:00.108090 1132 raft_consensus.cc:348] T ab995d6a8aeb42218230cf4f7cda448b P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:00.108386 1132 raft_consensus.cc:374] T ab995d6a8aeb42218230cf4f7cda448b P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.108502 1132 raft_consensus.cc:729] T ab995d6a8aeb42218230cf4f7cda448b P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.108826 1132 consensus_queue.cc:260] T ab995d6a8aeb42218230cf4f7cda448b P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:00.111948 1137 tablet_bootstrap.cc:492] T b7b284cffcb04ce883a24f6a8796748f P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:00.112205 1137 ts_tablet_manager.cc:1367] T b7b284cffcb04ce883a24f6a8796748f P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.084s user 0.023s sys 0.028s
I20240503 02:13:00.112993 1137 raft_consensus.cc:348] T b7b284cffcb04ce883a24f6a8796748f P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:00.113358 1137 raft_consensus.cc:374] T b7b284cffcb04ce883a24f6a8796748f P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.113452 1137 raft_consensus.cc:729] T b7b284cffcb04ce883a24f6a8796748f P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.113742 1137 consensus_queue.cc:260] T b7b284cffcb04ce883a24f6a8796748f P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:00.114356 1137 ts_tablet_manager.cc:1398] T b7b284cffcb04ce883a24f6a8796748f P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:13:00.114584 1131 raft_consensus.cc:348] T ab995d6a8aeb42218230cf4f7cda448b P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:00.114975 1131 raft_consensus.cc:374] T ab995d6a8aeb42218230cf4f7cda448b P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.115298 1131 raft_consensus.cc:729] T ab995d6a8aeb42218230cf4f7cda448b P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.115667 1131 consensus_queue.cc:260] T ab995d6a8aeb42218230cf4f7cda448b P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:00.115950 1137 tablet_bootstrap.cc:492] T f82e505ac13a41019e422b9dc1e4fef9 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:00.117640 1137 tablet_bootstrap.cc:654] T f82e505ac13a41019e422b9dc1e4fef9 P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.129662 1131 ts_tablet_manager.cc:1398] T ab995d6a8aeb42218230cf4f7cda448b P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.174s user 0.001s sys 0.001s
I20240503 02:13:00.137112 32621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:00.139942 643 raft_consensus.cc:482] T b7b284cffcb04ce883a24f6a8796748f P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:00.140125 643 raft_consensus.cc:504] T b7b284cffcb04ce883a24f6a8796748f P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:00.140746 643 leader_election.cc:288] T b7b284cffcb04ce883a24f6a8796748f P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
W20240503 02:13:00.142036 32709 leader_election.cc:341] T b7b284cffcb04ce883a24f6a8796748f P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Not found: Tablet not found: b7b284cffcb04ce883a24f6a8796748f
I20240503 02:13:00.142339 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7b284cffcb04ce883a24f6a8796748f" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
W20240503 02:13:00.142923 32709 leader_election.cc:341] T b7b284cffcb04ce883a24f6a8796748f P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:00.143055 32709 leader_election.cc:302] T b7b284cffcb04ce883a24f6a8796748f P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead; no voters: 9a4d92bc25374fb3960fde7d10be309d, fbbc2be6382e47908ecb7c68c367bd26
I20240503 02:13:00.143455 643 raft_consensus.cc:2726] T b7b284cffcb04ce883a24f6a8796748f P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:00.157169 29731 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240503 02:13:00.157450 29731 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240503 02:13:00.158311 1137 tablet_bootstrap.cc:492] T f82e505ac13a41019e422b9dc1e4fef9 P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:00.158581 1137 ts_tablet_manager.cc:1367] T f82e505ac13a41019e422b9dc1e4fef9 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.043s user 0.009s sys 0.030s
I20240503 02:13:00.160043 1137 raft_consensus.cc:348] T f82e505ac13a41019e422b9dc1e4fef9 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:00.160326 1137 raft_consensus.cc:374] T f82e505ac13a41019e422b9dc1e4fef9 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.160437 1137 raft_consensus.cc:729] T f82e505ac13a41019e422b9dc1e4fef9 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.160786 1137 consensus_queue.cc:260] T f82e505ac13a41019e422b9dc1e4fef9 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:00.161409 1137 ts_tablet_manager.cc:1398] T f82e505ac13a41019e422b9dc1e4fef9 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:13:00.161688 29731 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet ed32c167c0d74ee588b59f240d2e5b24
I20240503 02:13:00.162253 29731 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet 4de7cb0be278476481d01083add5c978
I20240503 02:13:00.162636 29731 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet c3f04e7716c54d6090a00d7a6b771b64
I20240503 02:13:00.162788 29853 tablet_service.cc:1514] Processing DeleteTablet for tablet ed32c167c0d74ee588b59f240d2e5b24 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:00 UTC) from {username='slave'} at 127.0.0.1:58744
I20240503 02:13:00.163066 980 tablet_replica.cc:330] T ed32c167c0d74ee588b59f240d2e5b24 P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:13:00.163249 980 raft_consensus.cc:2229] T ed32c167c0d74ee588b59f240d2e5b24 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:00.163478 980 raft_consensus.cc:2258] T ed32c167c0d74ee588b59f240d2e5b24 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:00.163578 30114 tablet_service.cc:1514] Processing DeleteTablet for tablet ed32c167c0d74ee588b59f240d2e5b24 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:00 UTC) from {username='slave'} at 127.0.0.1:44812
I20240503 02:13:00.163866 1024 tablet_replica.cc:330] T ed32c167c0d74ee588b59f240d2e5b24 P a9094240860e43bfb129193f3bfd5be6: stopping tablet replica
I20240503 02:13:00.164023 29986 tablet_service.cc:1514] Processing DeleteTablet for tablet ed32c167c0d74ee588b59f240d2e5b24 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:00 UTC) from {username='slave'} at 127.0.0.1:52158
I20240503 02:13:00.164104 1024 raft_consensus.cc:2229] T ed32c167c0d74ee588b59f240d2e5b24 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:00.164253 980 ts_tablet_manager.cc:1875] T ed32c167c0d74ee588b59f240d2e5b24 P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:00.164355 1024 raft_consensus.cc:2258] T ed32c167c0d74ee588b59f240d2e5b24 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:00.165115 1024 ts_tablet_manager.cc:1875] T ed32c167c0d74ee588b59f240d2e5b24 P a9094240860e43bfb129193f3bfd5be6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:00.166209 30244 tablet_service.cc:1514] Processing DeleteTablet for tablet 4de7cb0be278476481d01083add5c978 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:00 UTC) from {username='slave'} at 127.0.0.1:49856
I20240503 02:13:00.166457 30245 tablet_service.cc:1514] Processing DeleteTablet for tablet c3f04e7716c54d6090a00d7a6b771b64 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:00 UTC) from {username='slave'} at 127.0.0.1:49856
I20240503 02:13:00.168133 29853 tablet_service.cc:1514] Processing DeleteTablet for tablet 4de7cb0be278476481d01083add5c978 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:00 UTC) from {username='slave'} at 127.0.0.1:58744
I20240503 02:13:00.168469 657 tablet_replica.cc:330] T ed32c167c0d74ee588b59f240d2e5b24 P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:13:00.168728 657 raft_consensus.cc:2229] T ed32c167c0d74ee588b59f240d2e5b24 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:00.168816 29984 tablet_service.cc:1514] Processing DeleteTablet for tablet c3f04e7716c54d6090a00d7a6b771b64 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:00 UTC) from {username='slave'} at 127.0.0.1:52158
I20240503 02:13:00.169101 657 raft_consensus.cc:2258] T ed32c167c0d74ee588b59f240d2e5b24 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:00.169646 29986 tablet_service.cc:1514] Processing DeleteTablet for tablet 4de7cb0be278476481d01083add5c978 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:00 UTC) from {username='slave'} at 127.0.0.1:52158
I20240503 02:13:00.169814 657 ts_tablet_manager.cc:1875] T ed32c167c0d74ee588b59f240d2e5b24 P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:00.170734 980 ts_tablet_manager.cc:1888] T ed32c167c0d74ee588b59f240d2e5b24 P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:13:00.170866 980 log.cc:1196] T ed32c167c0d74ee588b59f240d2e5b24 P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/ed32c167c0d74ee588b59f240d2e5b24
I20240503 02:13:00.171332 980 ts_tablet_manager.cc:1909] T ed32c167c0d74ee588b59f240d2e5b24 P 16440e365430405fb10366ab70e2b429: Deleting consensus metadata
I20240503 02:13:00.172233 980 tablet_replica.cc:330] T 4de7cb0be278476481d01083add5c978 P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:13:00.172331 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet ed32c167c0d74ee588b59f240d2e5b24 (table rebalance_test_table_16 [id=0cc2eea0038e4971a3d7119cbaabc735]) successfully deleted
I20240503 02:13:00.172427 980 raft_consensus.cc:2229] T 4de7cb0be278476481d01083add5c978 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:00.172642 980 raft_consensus.cc:2258] T 4de7cb0be278476481d01083add5c978 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:00.174363 656 tablet_replica.cc:330] T 4de7cb0be278476481d01083add5c978 P 1f10f80f13c84e80b03eda7085a3b012: stopping tablet replica
I20240503 02:13:00.174588 656 raft_consensus.cc:2229] T 4de7cb0be278476481d01083add5c978 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:00.174832 656 raft_consensus.cc:2258] T 4de7cb0be278476481d01083add5c978 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:00.175530 656 ts_tablet_manager.cc:1875] T 4de7cb0be278476481d01083add5c978 P 1f10f80f13c84e80b03eda7085a3b012: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:00.176025 657 ts_tablet_manager.cc:1888] T ed32c167c0d74ee588b59f240d2e5b24 P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:13:00.176146 657 log.cc:1196] T ed32c167c0d74ee588b59f240d2e5b24 P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/ed32c167c0d74ee588b59f240d2e5b24
I20240503 02:13:00.176573 657 ts_tablet_manager.cc:1909] T ed32c167c0d74ee588b59f240d2e5b24 P 155735c7f8b14997ac4c770aee2a9401: Deleting consensus metadata
I20240503 02:13:00.176625 1112 tablet_bootstrap.cc:492] T 35a5a7f4fe184b9fadc94c98b61deaf0 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:00.177116 1112 tablet_bootstrap.cc:492] T 35a5a7f4fe184b9fadc94c98b61deaf0 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap complete.
I20240503 02:13:00.177392 1112 ts_tablet_manager.cc:1367] T 35a5a7f4fe184b9fadc94c98b61deaf0 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.175s user 0.062s sys 0.004s
I20240503 02:13:00.177687 657 tablet_replica.cc:330] T c3f04e7716c54d6090a00d7a6b771b64 P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:13:00.177942 657 raft_consensus.cc:2229] T c3f04e7716c54d6090a00d7a6b771b64 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:00.178225 657 raft_consensus.cc:2258] T c3f04e7716c54d6090a00d7a6b771b64 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:00.178792 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet ed32c167c0d74ee588b59f240d2e5b24 (table rebalance_test_table_16 [id=0cc2eea0038e4971a3d7119cbaabc735]) successfully deleted
I20240503 02:13:00.179068 1136 tablet_bootstrap.cc:492] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:00.179105 657 ts_tablet_manager.cc:1875] T c3f04e7716c54d6090a00d7a6b771b64 P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:00.179198 1136 ts_tablet_manager.cc:1367] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.184s user 0.019s sys 0.048s
I20240503 02:13:00.180236 1135 tablet_bootstrap.cc:654] T f82e505ac13a41019e422b9dc1e4fef9 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.180264 1132 ts_tablet_manager.cc:1398] T ab995d6a8aeb42218230cf4f7cda448b P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.073s user 0.002s sys 0.000s
I20240503 02:13:00.181264 30114 tablet_service.cc:1514] Processing DeleteTablet for tablet c3f04e7716c54d6090a00d7a6b771b64 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:00 UTC) from {username='slave'} at 127.0.0.1:44812
I20240503 02:13:00.181711 980 ts_tablet_manager.cc:1875] T 4de7cb0be278476481d01083add5c978 P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:00.178174 1112 raft_consensus.cc:348] T 35a5a7f4fe184b9fadc94c98b61deaf0 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:00.182036 1112 raft_consensus.cc:729] T 35a5a7f4fe184b9fadc94c98b61deaf0 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.182351 1136 raft_consensus.cc:348] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:00.182551 1136 raft_consensus.cc:374] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.182643 1136 raft_consensus.cc:729] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.182868 1136 consensus_queue.cc:260] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:00.183439 1136 ts_tablet_manager.cc:1398] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 02:13:00.183668 1112 consensus_queue.cc:260] T 35a5a7f4fe184b9fadc94c98b61deaf0 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:00.185531 1112 ts_tablet_manager.cc:1398] T 35a5a7f4fe184b9fadc94c98b61deaf0 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240503 02:13:00.186865 657 ts_tablet_manager.cc:1888] T c3f04e7716c54d6090a00d7a6b771b64 P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:13:00.187079 657 log.cc:1196] T c3f04e7716c54d6090a00d7a6b771b64 P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/c3f04e7716c54d6090a00d7a6b771b64
I20240503 02:13:00.187677 657 ts_tablet_manager.cc:1909] T c3f04e7716c54d6090a00d7a6b771b64 P 155735c7f8b14997ac4c770aee2a9401: Deleting consensus metadata
I20240503 02:13:00.188035 1132 tablet_bootstrap.cc:492] T b7b284cffcb04ce883a24f6a8796748f P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:00.188661 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet c3f04e7716c54d6090a00d7a6b771b64 (table rebalance_test_table_16 [id=0cc2eea0038e4971a3d7119cbaabc735]) successfully deleted
I20240503 02:13:00.190256 657 tablet_replica.cc:330] T 4de7cb0be278476481d01083add5c978 P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:13:00.190505 657 raft_consensus.cc:2229] T 4de7cb0be278476481d01083add5c978 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:00.190862 657 raft_consensus.cc:2258] T 4de7cb0be278476481d01083add5c978 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:00.191630 657 ts_tablet_manager.cc:1875] T 4de7cb0be278476481d01083add5c978 P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:00.192739 656 ts_tablet_manager.cc:1888] T 4de7cb0be278476481d01083add5c978 P 1f10f80f13c84e80b03eda7085a3b012: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:13:00.192940 656 log.cc:1196] T 4de7cb0be278476481d01083add5c978 P 1f10f80f13c84e80b03eda7085a3b012: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/wals/4de7cb0be278476481d01083add5c978
I20240503 02:13:00.194195 1132 tablet_bootstrap.cc:654] T b7b284cffcb04ce883a24f6a8796748f P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.197326 1024 ts_tablet_manager.cc:1888] T ed32c167c0d74ee588b59f240d2e5b24 P a9094240860e43bfb129193f3bfd5be6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:13:00.197485 1024 log.cc:1196] T ed32c167c0d74ee588b59f240d2e5b24 P a9094240860e43bfb129193f3bfd5be6: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/wals/ed32c167c0d74ee588b59f240d2e5b24
I20240503 02:13:00.197901 1024 ts_tablet_manager.cc:1909] T ed32c167c0d74ee588b59f240d2e5b24 P a9094240860e43bfb129193f3bfd5be6: Deleting consensus metadata
I20240503 02:13:00.198917 657 ts_tablet_manager.cc:1888] T 4de7cb0be278476481d01083add5c978 P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:13:00.198974 29718 catalog_manager.cc:4830] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): tablet ed32c167c0d74ee588b59f240d2e5b24 (table rebalance_test_table_16 [id=0cc2eea0038e4971a3d7119cbaabc735]) successfully deleted
I20240503 02:13:00.199048 657 log.cc:1196] T 4de7cb0be278476481d01083add5c978 P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/4de7cb0be278476481d01083add5c978
I20240503 02:13:00.199529 657 ts_tablet_manager.cc:1909] T 4de7cb0be278476481d01083add5c978 P 155735c7f8b14997ac4c770aee2a9401: Deleting consensus metadata
I20240503 02:13:00.200480 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet 4de7cb0be278476481d01083add5c978 (table rebalance_test_table_16 [id=0cc2eea0038e4971a3d7119cbaabc735]) successfully deleted
I20240503 02:13:00.201020 656 ts_tablet_manager.cc:1909] T 4de7cb0be278476481d01083add5c978 P 1f10f80f13c84e80b03eda7085a3b012: Deleting consensus metadata
I20240503 02:13:00.202065 29719 catalog_manager.cc:4830] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): tablet 4de7cb0be278476481d01083add5c978 (table rebalance_test_table_16 [id=0cc2eea0038e4971a3d7119cbaabc735]) successfully deleted
I20240503 02:13:00.202181 656 tablet_replica.cc:330] T c3f04e7716c54d6090a00d7a6b771b64 P 1f10f80f13c84e80b03eda7085a3b012: stopping tablet replica
I20240503 02:13:00.202363 656 raft_consensus.cc:2229] T c3f04e7716c54d6090a00d7a6b771b64 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:00.202699 656 raft_consensus.cc:2258] T c3f04e7716c54d6090a00d7a6b771b64 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:00.203003 1024 tablet_replica.cc:330] T c3f04e7716c54d6090a00d7a6b771b64 P a9094240860e43bfb129193f3bfd5be6: stopping tablet replica
I20240503 02:13:00.203248 1024 raft_consensus.cc:2229] T c3f04e7716c54d6090a00d7a6b771b64 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:00.203459 656 ts_tablet_manager.cc:1875] T c3f04e7716c54d6090a00d7a6b771b64 P 1f10f80f13c84e80b03eda7085a3b012: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:00.207176 1024 raft_consensus.cc:2258] T c3f04e7716c54d6090a00d7a6b771b64 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:00.207715 1135 tablet_bootstrap.cc:492] T f82e505ac13a41019e422b9dc1e4fef9 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:13:00.207872 1135 ts_tablet_manager.cc:1367] T f82e505ac13a41019e422b9dc1e4fef9 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.137s user 0.000s sys 0.010s
I20240503 02:13:00.208700 1135 raft_consensus.cc:348] T f82e505ac13a41019e422b9dc1e4fef9 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:00.209460 1136 tablet_bootstrap.cc:492] T 9102ab9545c54e82b642673795cb747a P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:00.209583 30026 tablet_copy_service.cc:342] P 155735c7f8b14997ac4c770aee2a9401: Request end of tablet copy session fbbc2be6382e47908ecb7c68c367bd26-35a5a7f4fe184b9fadc94c98b61deaf0 received from {username='slave'} at 127.22.108.7:60369
I20240503 02:13:00.209743 30026 tablet_copy_service.cc:433] P 155735c7f8b14997ac4c770aee2a9401: ending tablet copy session fbbc2be6382e47908ecb7c68c367bd26-35a5a7f4fe184b9fadc94c98b61deaf0 on tablet 35a5a7f4fe184b9fadc94c98b61deaf0 with peer fbbc2be6382e47908ecb7c68c367bd26
I20240503 02:13:00.211129 656 ts_tablet_manager.cc:1888] T c3f04e7716c54d6090a00d7a6b771b64 P 1f10f80f13c84e80b03eda7085a3b012: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:13:00.211242 1024 ts_tablet_manager.cc:1875] T c3f04e7716c54d6090a00d7a6b771b64 P a9094240860e43bfb129193f3bfd5be6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:00.211347 656 log.cc:1196] T c3f04e7716c54d6090a00d7a6b771b64 P 1f10f80f13c84e80b03eda7085a3b012: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/wals/c3f04e7716c54d6090a00d7a6b771b64
I20240503 02:13:00.211907 656 ts_tablet_manager.cc:1909] T c3f04e7716c54d6090a00d7a6b771b64 P 1f10f80f13c84e80b03eda7085a3b012: Deleting consensus metadata
I20240503 02:13:00.212951 1136 tablet_bootstrap.cc:654] T 9102ab9545c54e82b642673795cb747a P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.213003 29719 catalog_manager.cc:4830] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): tablet c3f04e7716c54d6090a00d7a6b771b64 (table rebalance_test_table_16 [id=0cc2eea0038e4971a3d7119cbaabc735]) successfully deleted
I20240503 02:13:00.216861 1024 ts_tablet_manager.cc:1888] T c3f04e7716c54d6090a00d7a6b771b64 P a9094240860e43bfb129193f3bfd5be6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:13:00.217008 1024 log.cc:1196] T c3f04e7716c54d6090a00d7a6b771b64 P a9094240860e43bfb129193f3bfd5be6: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/wals/c3f04e7716c54d6090a00d7a6b771b64
I20240503 02:13:00.217417 1024 ts_tablet_manager.cc:1909] T c3f04e7716c54d6090a00d7a6b771b64 P a9094240860e43bfb129193f3bfd5be6: Deleting consensus metadata
I20240503 02:13:00.217810 1135 raft_consensus.cc:374] T f82e505ac13a41019e422b9dc1e4fef9 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.217931 1135 raft_consensus.cc:729] T f82e505ac13a41019e422b9dc1e4fef9 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.218221 1135 consensus_queue.cc:260] T f82e505ac13a41019e422b9dc1e4fef9 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:00.218815 1135 ts_tablet_manager.cc:1398] T f82e505ac13a41019e422b9dc1e4fef9 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240503 02:13:00.219246 980 ts_tablet_manager.cc:1888] T 4de7cb0be278476481d01083add5c978 P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:13:00.219429 980 log.cc:1196] T 4de7cb0be278476481d01083add5c978 P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/4de7cb0be278476481d01083add5c978
I20240503 02:13:00.219869 980 ts_tablet_manager.cc:1909] T 4de7cb0be278476481d01083add5c978 P 16440e365430405fb10366ab70e2b429: Deleting consensus metadata
I20240503 02:13:00.221532 1132 tablet_bootstrap.cc:492] T b7b284cffcb04ce883a24f6a8796748f P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:00.221781 1132 ts_tablet_manager.cc:1367] T b7b284cffcb04ce883a24f6a8796748f P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.000s
I20240503 02:13:00.220185 1135 tablet_bootstrap.cc:492] T 9102ab9545c54e82b642673795cb747a P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:13:00.226212 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet 4de7cb0be278476481d01083add5c978 (table rebalance_test_table_16 [id=0cc2eea0038e4971a3d7119cbaabc735]) successfully deleted
I20240503 02:13:00.227190 1132 raft_consensus.cc:348] T b7b284cffcb04ce883a24f6a8796748f P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:00.227403 1135 tablet_bootstrap.cc:654] T 9102ab9545c54e82b642673795cb747a P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.227555 1132 raft_consensus.cc:374] T b7b284cffcb04ce883a24f6a8796748f P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.227726 1132 raft_consensus.cc:729] T b7b284cffcb04ce883a24f6a8796748f P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.228103 1132 consensus_queue.cc:260] T b7b284cffcb04ce883a24f6a8796748f P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:00.228606 29718 catalog_manager.cc:4830] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): tablet c3f04e7716c54d6090a00d7a6b771b64 (table rebalance_test_table_16 [id=0cc2eea0038e4971a3d7119cbaabc735]) successfully deleted
I20240503 02:13:00.230082 1132 ts_tablet_manager.cc:1398] T b7b284cffcb04ce883a24f6a8796748f P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:13:00.236934 1136 tablet_bootstrap.cc:492] T 9102ab9545c54e82b642673795cb747a P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:00.239483 1136 ts_tablet_manager.cc:1367] T 9102ab9545c54e82b642673795cb747a P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.004s
I20240503 02:13:00.241521 436 raft_consensus.cc:1203] T 35a5a7f4fe184b9fadc94c98b61deaf0 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:00.242488 1136 raft_consensus.cc:348] T 9102ab9545c54e82b642673795cb747a P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:00.242681 1136 raft_consensus.cc:374] T 9102ab9545c54e82b642673795cb747a P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.242770 1136 raft_consensus.cc:729] T 9102ab9545c54e82b642673795cb747a P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.243069 1136 consensus_queue.cc:260] T 9102ab9545c54e82b642673795cb747a P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:00.243781 1136 ts_tablet_manager.cc:1398] T 9102ab9545c54e82b642673795cb747a P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:13:00.244184 1136 tablet_bootstrap.cc:492] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:00.246994 1136 tablet_bootstrap.cc:654] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.250026 1135 tablet_bootstrap.cc:492] T 9102ab9545c54e82b642673795cb747a P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:13:00.250253 1135 ts_tablet_manager.cc:1367] T 9102ab9545c54e82b642673795cb747a P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.030s user 0.015s sys 0.008s
I20240503 02:13:00.251137 1135 raft_consensus.cc:348] T 9102ab9545c54e82b642673795cb747a P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:00.251400 1135 raft_consensus.cc:374] T 9102ab9545c54e82b642673795cb747a P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.251571 1135 raft_consensus.cc:729] T 9102ab9545c54e82b642673795cb747a P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.251878 1135 consensus_queue.cc:260] T 9102ab9545c54e82b642673795cb747a P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:00.252486 1135 ts_tablet_manager.cc:1398] T 9102ab9545c54e82b642673795cb747a P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:13:00.273576 1136 tablet_bootstrap.cc:492] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:00.273777 1136 ts_tablet_manager.cc:1367] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20240503 02:13:00.274395 1136 raft_consensus.cc:348] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:00.274590 1136 raft_consensus.cc:374] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.274674 1136 raft_consensus.cc:729] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.274874 1136 consensus_queue.cc:260] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:00.275344 1136 ts_tablet_manager.cc:1398] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:13:00.295320 1042 raft_consensus.cc:482] T 9102ab9545c54e82b642673795cb747a P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:00.295538 1042 raft_consensus.cc:504] T 9102ab9545c54e82b642673795cb747a P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:00.296236 1042 leader_election.cc:288] T 9102ab9545c54e82b642673795cb747a P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:13:00.297353 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9102ab9545c54e82b642673795cb747a" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:13:00.297583 30004 raft_consensus.cc:2454] T 9102ab9545c54e82b642673795cb747a P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:13:00.298270 29809 leader_election.cc:302] T 9102ab9545c54e82b642673795cb747a P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429; no voters:
I20240503 02:13:00.298702 1042 raft_consensus.cc:2781] T 9102ab9545c54e82b642673795cb747a P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:00.298907 1042 raft_consensus.cc:482] T 9102ab9545c54e82b642673795cb747a P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:00.299093 1042 raft_consensus.cc:3037] T 9102ab9545c54e82b642673795cb747a P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:00.302855 1042 raft_consensus.cc:504] T 9102ab9545c54e82b642673795cb747a P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:00.303534 1042 leader_election.cc:288] T 9102ab9545c54e82b642673795cb747a P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Requested vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:13:00.304065 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9102ab9545c54e82b642673795cb747a" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:13:00.304250 30004 raft_consensus.cc:3037] T 9102ab9545c54e82b642673795cb747a P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:00.304641 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9102ab9545c54e82b642673795cb747a" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:13:00.304705 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9102ab9545c54e82b642673795cb747a" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:13:00.304862 436 raft_consensus.cc:2454] T 9102ab9545c54e82b642673795cb747a P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:13:00.306501 30004 raft_consensus.cc:2454] T 9102ab9545c54e82b642673795cb747a P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:13:00.306989 29809 leader_election.cc:302] T 9102ab9545c54e82b642673795cb747a P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429; no voters: fbbc2be6382e47908ecb7c68c367bd26
I20240503 02:13:00.307375 1042 raft_consensus.cc:2781] T 9102ab9545c54e82b642673795cb747a P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:00.307549 1042 raft_consensus.cc:686] T 9102ab9545c54e82b642673795cb747a P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Becoming Leader. State: Replica: 16440e365430405fb10366ab70e2b429, State: Running, Role: LEADER
I20240503 02:13:00.307888 1042 consensus_queue.cc:237] T 9102ab9545c54e82b642673795cb747a P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:00.309988 778 raft_consensus.cc:482] T ab995d6a8aeb42218230cf4f7cda448b P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:00.310146 778 raft_consensus.cc:504] T ab995d6a8aeb42218230cf4f7cda448b P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:00.310964 778 leader_election.cc:288] T ab995d6a8aeb42218230cf4f7cda448b P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:13:00.311344 568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab995d6a8aeb42218230cf4f7cda448b" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:00.311596 568 raft_consensus.cc:2454] T ab995d6a8aeb42218230cf4f7cda448b P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:13:00.312080 29939 leader_election.cc:302] T ab995d6a8aeb42218230cf4f7cda448b P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:00.312409 778 raft_consensus.cc:2781] T ab995d6a8aeb42218230cf4f7cda448b P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:00.312536 778 raft_consensus.cc:482] T ab995d6a8aeb42218230cf4f7cda448b P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:00.312640 778 raft_consensus.cc:3037] T ab995d6a8aeb42218230cf4f7cda448b P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:00.312789 29725 catalog_manager.cc:5484] T 9102ab9545c54e82b642673795cb747a P 16440e365430405fb10366ab70e2b429 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16440e365430405fb10366ab70e2b429 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:00.315275 778 raft_consensus.cc:504] T ab995d6a8aeb42218230cf4f7cda448b P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:00.315869 778 leader_election.cc:288] T ab995d6a8aeb42218230cf4f7cda448b P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Requested vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:13:00.316741 568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab995d6a8aeb42218230cf4f7cda448b" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:13:00.316929 568 raft_consensus.cc:3037] T ab995d6a8aeb42218230cf4f7cda448b P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:00.318993 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab995d6a8aeb42218230cf4f7cda448b" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:13:00.319070 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab995d6a8aeb42218230cf4f7cda448b" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d"
I20240503 02:13:00.319231 32641 raft_consensus.cc:2454] T ab995d6a8aeb42218230cf4f7cda448b P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:13:00.319613 568 raft_consensus.cc:2454] T ab995d6a8aeb42218230cf4f7cda448b P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:13:00.320369 29939 leader_election.cc:302] T ab995d6a8aeb42218230cf4f7cda448b P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, f575ce92c0224eeb80b9fae5f6561fbe; no voters: 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:13:00.320636 778 raft_consensus.cc:2781] T ab995d6a8aeb42218230cf4f7cda448b P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:00.320775 778 raft_consensus.cc:686] T ab995d6a8aeb42218230cf4f7cda448b P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Becoming Leader. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Running, Role: LEADER
I20240503 02:13:00.321049 778 consensus_queue.cc:237] T ab995d6a8aeb42218230cf4f7cda448b P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:00.324321 29725 catalog_manager.cc:5484] T ab995d6a8aeb42218230cf4f7cda448b P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: term changed from 0 to 1, leader changed from <none> to 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2). New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:00.362771 30266 consensus_queue.cc:237] T 0ae1063008d44f6eaa61a6e353732e48 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:00.365862 30136 raft_consensus.cc:1261] T 0ae1063008d44f6eaa61a6e353732e48 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:00.366465 1143 consensus_queue.cc:1035] T 0ae1063008d44f6eaa61a6e353732e48 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:00.375104 30004 raft_consensus.cc:1261] T 0ae1063008d44f6eaa61a6e353732e48 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:00.375782 1143 consensus_queue.cc:1035] T 0ae1063008d44f6eaa61a6e353732e48 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:00.376791 762 raft_consensus.cc:482] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:00.377007 762 raft_consensus.cc:504] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:00.377787 762 leader_election.cc:288] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:00.378391 643 raft_consensus.cc:482] T f82e505ac13a41019e422b9dc1e4fef9 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:00.378489 29872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f82e505ac13a41019e422b9dc1e4fef9" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:13:00.378574 643 raft_consensus.cc:504] T f82e505ac13a41019e422b9dc1e4fef9 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:00.378746 29872 raft_consensus.cc:2454] T f82e505ac13a41019e422b9dc1e4fef9 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 0.
I20240503 02:13:00.379382 643 leader_election.cc:288] T f82e505ac13a41019e422b9dc1e4fef9 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:00.379829 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f82e505ac13a41019e422b9dc1e4fef9" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:13:00.378602 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f82e505ac13a41019e422b9dc1e4fef9" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:13:00.380044 436 raft_consensus.cc:2454] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:13:00.380164 305 raft_consensus.cc:2454] T f82e505ac13a41019e422b9dc1e4fef9 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 0.
I20240503 02:13:00.380045 373 leader_election.cc:302] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:00.381130 32709 leader_election.cc:302] T f82e505ac13a41019e422b9dc1e4fef9 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:00.381301 29872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f82e505ac13a41019e422b9dc1e4fef9" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:13:00.381481 29872 raft_consensus.cc:2454] T f82e505ac13a41019e422b9dc1e4fef9 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:13:00.381862 643 raft_consensus.cc:2781] T f82e505ac13a41019e422b9dc1e4fef9 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:00.381953 762 raft_consensus.cc:2781] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:00.381974 643 raft_consensus.cc:482] T f82e505ac13a41019e422b9dc1e4fef9 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:00.382079 643 raft_consensus.cc:3037] T f82e505ac13a41019e422b9dc1e4fef9 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:00.382042 762 raft_consensus.cc:482] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:00.382162 762 raft_consensus.cc:3037] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:00.385011 762 raft_consensus.cc:504] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
W20240503 02:13:00.385890 30200 consensus_peers.cc:485] T 0ae1063008d44f6eaa61a6e353732e48 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0ae1063008d44f6eaa61a6e353732e48. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:00.386103 29872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f82e505ac13a41019e422b9dc1e4fef9" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:13:00.386289 29872 raft_consensus.cc:3037] T f82e505ac13a41019e422b9dc1e4fef9 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:00.386423 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f82e505ac13a41019e422b9dc1e4fef9" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:13:00.386670 762 leader_election.cc:288] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Requested vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:00.389796 643 raft_consensus.cc:504] T f82e505ac13a41019e422b9dc1e4fef9 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:00.390239 305 raft_consensus.cc:2379] T f82e505ac13a41019e422b9dc1e4fef9 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fbbc2be6382e47908ecb7c68c367bd26 in current term 1: Already voted for candidate 8a666e798fe8477d8dd1e51b385d5ead in this term.
I20240503 02:13:00.390532 643 leader_election.cc:288] T f82e505ac13a41019e422b9dc1e4fef9 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Requested vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:00.391767 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f82e505ac13a41019e422b9dc1e4fef9" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:13:00.392021 436 raft_consensus.cc:2379] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a666e798fe8477d8dd1e51b385d5ead in current term 1: Already voted for candidate fbbc2be6382e47908ecb7c68c367bd26 in this term.
I20240503 02:13:00.391501 1143 raft_consensus.cc:2932] T 0ae1063008d44f6eaa61a6e353732e48 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:00.391929 30006 raft_consensus.cc:2932] T 0ae1063008d44f6eaa61a6e353732e48 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:00.398134 30136 raft_consensus.cc:2932] T 0ae1063008d44f6eaa61a6e353732e48 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:00.399199 29725 catalog_manager.cc:5484] T 0ae1063008d44f6eaa61a6e353732e48 P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: config changed from index -1 to 4, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:00.402273 29872 raft_consensus.cc:2454] T f82e505ac13a41019e422b9dc1e4fef9 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 1.
I20240503 02:13:00.402655 373 leader_election.cc:302] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16440e365430405fb10366ab70e2b429, fbbc2be6382e47908ecb7c68c367bd26; no voters: 8a666e798fe8477d8dd1e51b385d5ead
I20240503 02:13:00.403155 762 raft_consensus.cc:2781] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:00.403781 29874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f82e505ac13a41019e422b9dc1e4fef9" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:13:00.403997 29874 raft_consensus.cc:2379] T f82e505ac13a41019e422b9dc1e4fef9 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a666e798fe8477d8dd1e51b385d5ead in current term 1: Already voted for candidate fbbc2be6382e47908ecb7c68c367bd26 in this term.
I20240503 02:13:00.405970 32711 leader_election.cc:302] T f82e505ac13a41019e422b9dc1e4fef9 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead; no voters: 16440e365430405fb10366ab70e2b429, fbbc2be6382e47908ecb7c68c367bd26
I20240503 02:13:00.408528 643 raft_consensus.cc:2726] T f82e505ac13a41019e422b9dc1e4fef9 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 02:13:00.410262 762 raft_consensus.cc:686] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Becoming Leader. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Running, Role: LEADER
I20240503 02:13:00.410645 762 consensus_queue.cc:237] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:00.413831 29731 catalog_manager.cc:5484] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7). New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:00.474956 436 consensus_queue.cc:237] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:00.477669 568 raft_consensus.cc:1261] T 83c7d1060ccf49a2adda4302600a9876 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:00.478487 32641 raft_consensus.cc:1261] T 83c7d1060ccf49a2adda4302600a9876 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:00.478713 1200 consensus_queue.cc:1035] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:00.479171 762 consensus_queue.cc:1035] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:13:00.484597 371 consensus_peers.cc:485] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): Couldn't send request to peer 1f10f80f13c84e80b03eda7085a3b012. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 83c7d1060ccf49a2adda4302600a9876. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:00.486816 1200 raft_consensus.cc:482] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:00.487010 1200 raft_consensus.cc:504] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:00.486574 762 raft_consensus.cc:2932] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:00.487210 568 raft_consensus.cc:2932] T 83c7d1060ccf49a2adda4302600a9876 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:00.487746 1200 leader_election.cc:288] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:13:00.488713 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7b284cffcb04ce883a24f6a8796748f" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:13:00.488955 305 raft_consensus.cc:2454] T b7b284cffcb04ce883a24f6a8796748f P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 0.
I20240503 02:13:00.489598 371 leader_election.cc:302] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:00.489893 1200 raft_consensus.cc:2781] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:00.489992 1200 raft_consensus.cc:482] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:00.489960 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7b284cffcb04ce883a24f6a8796748f" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:13:00.490103 1200 raft_consensus.cc:3037] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:00.490150 32641 raft_consensus.cc:2454] T b7b284cffcb04ce883a24f6a8796748f P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 0.
I20240503 02:13:00.493139 29731 catalog_manager.cc:5484] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:00.493295 1200 raft_consensus.cc:504] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:00.493947 1200 leader_election.cc:288] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Requested vote from peers 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:13:00.491627 32641 raft_consensus.cc:2932] T 83c7d1060ccf49a2adda4302600a9876 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:00.495489 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7b284cffcb04ce883a24f6a8796748f" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:13:00.495699 305 raft_consensus.cc:3037] T b7b284cffcb04ce883a24f6a8796748f P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:00.498409 305 raft_consensus.cc:2454] T b7b284cffcb04ce883a24f6a8796748f P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 1.
I20240503 02:13:00.499802 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7b284cffcb04ce883a24f6a8796748f" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d"
I20240503 02:13:00.499887 371 leader_election.cc:302] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:00.500052 32641 raft_consensus.cc:3037] T b7b284cffcb04ce883a24f6a8796748f P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:00.500660 762 raft_consensus.cc:2781] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:00.500818 762 raft_consensus.cc:686] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Becoming Leader. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Running, Role: LEADER
I20240503 02:13:00.501123 762 consensus_queue.cc:237] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:00.503420 32641 raft_consensus.cc:2454] T b7b284cffcb04ce883a24f6a8796748f P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 1.
I20240503 02:13:00.505955 29731 catalog_manager.cc:5484] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7). New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:00.538122 29731 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:13:00.543817 29874 consensus_queue.cc:237] T 5f4b72b138c64ac58e6477cd2b1fb215 P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:00.544049 32641 raft_consensus.cc:3037] T ab995d6a8aeb42218230cf4f7cda448b P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:00.546044 568 raft_consensus.cc:1261] T ab995d6a8aeb42218230cf4f7cda448b P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:00.546591 986 consensus_queue.cc:1035] T ab995d6a8aeb42218230cf4f7cda448b P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:00.546660 32641 raft_consensus.cc:1261] T ab995d6a8aeb42218230cf4f7cda448b P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:00.547255 986 consensus_queue.cc:1035] T ab995d6a8aeb42218230cf4f7cda448b P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:00.551846 29725 catalog_manager.cc:5576] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595) does not have the latest schema for tablet ab995d6a8aeb42218230cf4f7cda448b (table rebalancer_dynamic_table [id=24bee9471db248d98a23cbedf87d8255]). Expected version 1 got 0
I20240503 02:13:00.555791 30006 raft_consensus.cc:1261] T 5f4b72b138c64ac58e6477cd2b1fb215 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:00.555804 30136 raft_consensus.cc:1261] T 5f4b72b138c64ac58e6477cd2b1fb215 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:00.556767 1042 consensus_queue.cc:1035] T 5f4b72b138c64ac58e6477cd2b1fb215 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:00.557343 702 consensus_queue.cc:1035] T 5f4b72b138c64ac58e6477cd2b1fb215 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:13:00.557633 29809 consensus_peers.cc:485] T 5f4b72b138c64ac58e6477cd2b1fb215 P 16440e365430405fb10366ab70e2b429 -> Peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): Couldn't send request to peer 8a666e798fe8477d8dd1e51b385d5ead. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f4b72b138c64ac58e6477cd2b1fb215. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:00.572468 29731 catalog_manager.cc:5576] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233) does not have the latest schema for tablet ab995d6a8aeb42218230cf4f7cda448b (table rebalancer_dynamic_table [id=24bee9471db248d98a23cbedf87d8255]). Expected version 1 got 0
I20240503 02:13:00.572477 1228 tablet.cc:1722] T ab995d6a8aeb42218230cf4f7cda448b P 9a4d92bc25374fb3960fde7d10be309d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:00.573084 29725 catalog_manager.cc:5576] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595) does not have the latest schema for tablet ab995d6a8aeb42218230cf4f7cda448b (table rebalancer_dynamic_table [id=24bee9471db248d98a23cbedf87d8255]). Expected version 1 got 0
I20240503 02:13:00.573712 1228 tablet.cc:1620] T ab995d6a8aeb42218230cf4f7cda448b P 9a4d92bc25374fb3960fde7d10be309d: MemRowSet was empty: no flush needed.
I20240503 02:13:00.573925 1227 tablet.cc:1722] T ab995d6a8aeb42218230cf4f7cda448b P 155735c7f8b14997ac4c770aee2a9401: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:00.574960 1227 tablet.cc:1620] T ab995d6a8aeb42218230cf4f7cda448b P 155735c7f8b14997ac4c770aee2a9401: MemRowSet was empty: no flush needed.
I20240503 02:13:00.577622 1042 raft_consensus.cc:2932] T 5f4b72b138c64ac58e6477cd2b1fb215 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:00.578302 30136 raft_consensus.cc:2932] T 5f4b72b138c64ac58e6477cd2b1fb215 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:00.579187 29731 catalog_manager.cc:5576] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233) does not have the latest schema for tablet ab995d6a8aeb42218230cf4f7cda448b (table rebalancer_dynamic_table [id=24bee9471db248d98a23cbedf87d8255]). Expected version 1 got 0
I20240503 02:13:00.582540 30006 raft_consensus.cc:2932] T 5f4b72b138c64ac58e6477cd2b1fb215 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:00.585206 1231 tablet.cc:1722] T ab995d6a8aeb42218230cf4f7cda448b P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:00.586723 29729 catalog_manager.cc:5484] T 5f4b72b138c64ac58e6477cd2b1fb215 P a9094240860e43bfb129193f3bfd5be6 reported cstate change: config changed from index -1 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:00.587924 1231 tablet.cc:1620] T ab995d6a8aeb42218230cf4f7cda448b P f575ce92c0224eeb80b9fae5f6561fbe: MemRowSet was empty: no flush needed.
I20240503 02:13:00.590418 29734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=24bee9471db248d98a23cbedf87d8255] alter complete (version 1)
I20240503 02:13:00.626758 30266 consensus_queue.cc:237] T 96de1c26ae884ea18b63b328db040ba1 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
W20240503 02:13:00.629345 30200 consensus_peers.cc:485] T 96de1c26ae884ea18b63b328db040ba1 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): Couldn't send request to peer fbbc2be6382e47908ecb7c68c367bd26. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96de1c26ae884ea18b63b328db040ba1. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:00.629360 29874 raft_consensus.cc:1261] T 96de1c26ae884ea18b63b328db040ba1 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:00.629691 30136 raft_consensus.cc:1261] T 96de1c26ae884ea18b63b328db040ba1 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:00.629890 1180 consensus_queue.cc:1035] T 96de1c26ae884ea18b63b328db040ba1 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:00.630156 1143 consensus_queue.cc:1035] T 96de1c26ae884ea18b63b328db040ba1 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:00.633496 1180 raft_consensus.cc:2932] T 96de1c26ae884ea18b63b328db040ba1 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:00.634057 29874 raft_consensus.cc:2932] T 96de1c26ae884ea18b63b328db040ba1 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:00.640112 29729 catalog_manager.cc:5484] T 96de1c26ae884ea18b63b328db040ba1 P 16440e365430405fb10366ab70e2b429 reported cstate change: config changed from index -1 to 4, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:00.640136 30134 raft_consensus.cc:2932] T 96de1c26ae884ea18b63b328db040ba1 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:00.678032 30134 consensus_queue.cc:237] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:00.679883 29731 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240503 02:13:00.680109 29731 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240503 02:13:00.681576 30266 raft_consensus.cc:1261] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:00.681576 30006 raft_consensus.cc:1261] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:00.682384 912 consensus_queue.cc:1035] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:00.682758 908 consensus_queue.cc:1035] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:00.684574 29731 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet dc27f3be29a9440c8b3eee46f04890d5
I20240503 02:13:00.685664 29853 tablet_service.cc:1514] Processing DeleteTablet for tablet dc27f3be29a9440c8b3eee46f04890d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:00 UTC) from {username='slave'} at 127.0.0.1:58744
I20240503 02:13:00.685827 29986 tablet_service.cc:1514] Processing DeleteTablet for tablet dc27f3be29a9440c8b3eee46f04890d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:00 UTC) from {username='slave'} at 127.0.0.1:52158
I20240503 02:13:00.685999 980 tablet_replica.cc:330] T dc27f3be29a9440c8b3eee46f04890d5 P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:13:00.686029 29731 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 4 replicas of tablet 0ae1063008d44f6eaa61a6e353732e48
I20240503 02:13:00.686115 30245 tablet_service.cc:1514] Processing DeleteTablet for tablet dc27f3be29a9440c8b3eee46f04890d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:00 UTC) from {username='slave'} at 127.0.0.1:49856
I20240503 02:13:00.686214 980 raft_consensus.cc:2229] T dc27f3be29a9440c8b3eee46f04890d5 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:00.686525 980 raft_consensus.cc:2258] T dc27f3be29a9440c8b3eee46f04890d5 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:00.686817 29986 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ae1063008d44f6eaa61a6e353732e48 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:00 UTC) from {username='slave'} at 127.0.0.1:52158
I20240503 02:13:00.687521 980 ts_tablet_manager.cc:1875] T dc27f3be29a9440c8b3eee46f04890d5 P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:00.688514 30245 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ae1063008d44f6eaa61a6e353732e48 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:00 UTC) from {username='slave'} at 127.0.0.1:49856
I20240503 02:13:00.689069 29731 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet 8fefbfb003914bf28c4176cd54a9130f
I20240503 02:13:00.689478 32620 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ae1063008d44f6eaa61a6e353732e48 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:00 UTC) from {username='slave'} at 127.0.0.1:38200
I20240503 02:13:00.689746 29986 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fefbfb003914bf28c4176cd54a9130f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:00 UTC) from {username='slave'} at 127.0.0.1:52158
I20240503 02:13:00.689915 656 tablet_replica.cc:330] T dc27f3be29a9440c8b3eee46f04890d5 P 1f10f80f13c84e80b03eda7085a3b012: stopping tablet replica
I20240503 02:13:00.690165 656 raft_consensus.cc:2229] T dc27f3be29a9440c8b3eee46f04890d5 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:00.690392 656 raft_consensus.cc:2258] T dc27f3be29a9440c8b3eee46f04890d5 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:00.691020 30245 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fefbfb003914bf28c4176cd54a9130f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:00 UTC) from {username='slave'} at 127.0.0.1:49856
I20240503 02:13:00.691246 656 ts_tablet_manager.cc:1875] T dc27f3be29a9440c8b3eee46f04890d5 P 1f10f80f13c84e80b03eda7085a3b012: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:00.693183 30114 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ae1063008d44f6eaa61a6e353732e48 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:00 UTC) from {username='slave'} at 127.0.0.1:44812
I20240503 02:13:00.693444 1024 tablet_replica.cc:330] T 0ae1063008d44f6eaa61a6e353732e48 P a9094240860e43bfb129193f3bfd5be6: stopping tablet replica
I20240503 02:13:00.693661 1024 raft_consensus.cc:2229] T 0ae1063008d44f6eaa61a6e353732e48 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:00.693920 1024 raft_consensus.cc:2258] T 0ae1063008d44f6eaa61a6e353732e48 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 02:13:00.694100 29719 catalog_manager.cc:4797] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): delete failed for tablet 0ae1063008d44f6eaa61a6e353732e48 because the tablet was not found. No further retry: Not found: Tablet not found: 0ae1063008d44f6eaa61a6e353732e48
I20240503 02:13:00.694890 1024 ts_tablet_manager.cc:1875] T 0ae1063008d44f6eaa61a6e353732e48 P a9094240860e43bfb129193f3bfd5be6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:00.689752 657 tablet_replica.cc:330] T dc27f3be29a9440c8b3eee46f04890d5 P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:13:00.696280 657 raft_consensus.cc:2229] T dc27f3be29a9440c8b3eee46f04890d5 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:00.696583 657 raft_consensus.cc:2258] T dc27f3be29a9440c8b3eee46f04890d5 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:00.697338 657 ts_tablet_manager.cc:1875] T dc27f3be29a9440c8b3eee46f04890d5 P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:00.697476 908 raft_consensus.cc:2932] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:00.698208 29731 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:13:00.697989 30266 raft_consensus.cc:2932] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:00.701447 29853 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fefbfb003914bf28c4176cd54a9130f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:00 UTC) from {username='slave'} at 127.0.0.1:58744
I20240503 02:13:00.702983 30006 raft_consensus.cc:2932] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:00.705376 29729 catalog_manager.cc:5484] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 reported cstate change: config changed from index -1 to 4, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:00.709758 1227 tablet.cc:1722] T ab995d6a8aeb42218230cf4f7cda448b P 155735c7f8b14997ac4c770aee2a9401: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:00.710238 1227 tablet.cc:1620] T ab995d6a8aeb42218230cf4f7cda448b P 155735c7f8b14997ac4c770aee2a9401: MemRowSet was empty: no flush needed.
I20240503 02:13:00.711541 1024 ts_tablet_manager.cc:1888] T 0ae1063008d44f6eaa61a6e353732e48 P a9094240860e43bfb129193f3bfd5be6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:00.711683 1024 log.cc:1196] T 0ae1063008d44f6eaa61a6e353732e48 P a9094240860e43bfb129193f3bfd5be6: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/wals/0ae1063008d44f6eaa61a6e353732e48
I20240503 02:13:00.712033 1024 ts_tablet_manager.cc:1909] T 0ae1063008d44f6eaa61a6e353732e48 P a9094240860e43bfb129193f3bfd5be6: Deleting consensus metadata
I20240503 02:13:00.712905 29718 catalog_manager.cc:4830] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): tablet 0ae1063008d44f6eaa61a6e353732e48 (table rebalance_test_table_18 [id=521f00c464174dec80c6db200c35e142]) successfully deleted
I20240503 02:13:00.716214 656 ts_tablet_manager.cc:1888] T dc27f3be29a9440c8b3eee46f04890d5 P 1f10f80f13c84e80b03eda7085a3b012: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:13:00.716387 656 log.cc:1196] T dc27f3be29a9440c8b3eee46f04890d5 P 1f10f80f13c84e80b03eda7085a3b012: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/wals/dc27f3be29a9440c8b3eee46f04890d5
I20240503 02:13:00.716881 656 ts_tablet_manager.cc:1909] T dc27f3be29a9440c8b3eee46f04890d5 P 1f10f80f13c84e80b03eda7085a3b012: Deleting consensus metadata
I20240503 02:13:00.717185 657 ts_tablet_manager.cc:1888] T dc27f3be29a9440c8b3eee46f04890d5 P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:13:00.717396 657 log.cc:1196] T dc27f3be29a9440c8b3eee46f04890d5 P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/dc27f3be29a9440c8b3eee46f04890d5
I20240503 02:13:00.717921 657 ts_tablet_manager.cc:1909] T dc27f3be29a9440c8b3eee46f04890d5 P 155735c7f8b14997ac4c770aee2a9401: Deleting consensus metadata
I20240503 02:13:00.718101 29719 catalog_manager.cc:4830] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): tablet dc27f3be29a9440c8b3eee46f04890d5 (table rebalance_test_table_18 [id=521f00c464174dec80c6db200c35e142]) successfully deleted
I20240503 02:13:00.719092 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet dc27f3be29a9440c8b3eee46f04890d5 (table rebalance_test_table_18 [id=521f00c464174dec80c6db200c35e142]) successfully deleted
I20240503 02:13:00.719043 1228 tablet.cc:1722] T ab995d6a8aeb42218230cf4f7cda448b P 9a4d92bc25374fb3960fde7d10be309d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:00.719569 1228 tablet.cc:1620] T ab995d6a8aeb42218230cf4f7cda448b P 9a4d92bc25374fb3960fde7d10be309d: MemRowSet was empty: no flush needed.
I20240503 02:13:00.719595 980 ts_tablet_manager.cc:1888] T dc27f3be29a9440c8b3eee46f04890d5 P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:13:00.719772 980 log.cc:1196] T dc27f3be29a9440c8b3eee46f04890d5 P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/dc27f3be29a9440c8b3eee46f04890d5
I20240503 02:13:00.720229 980 ts_tablet_manager.cc:1909] T dc27f3be29a9440c8b3eee46f04890d5 P 16440e365430405fb10366ab70e2b429: Deleting consensus metadata
I20240503 02:13:00.720438 657 tablet_replica.cc:330] T 0ae1063008d44f6eaa61a6e353732e48 P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:13:00.721488 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet dc27f3be29a9440c8b3eee46f04890d5 (table rebalance_test_table_18 [id=521f00c464174dec80c6db200c35e142]) successfully deleted
I20240503 02:13:00.721824 980 tablet_replica.cc:330] T 8fefbfb003914bf28c4176cd54a9130f P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:13:00.721873 657 raft_consensus.cc:2229] T 0ae1063008d44f6eaa61a6e353732e48 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:00.722136 980 raft_consensus.cc:2229] T 8fefbfb003914bf28c4176cd54a9130f P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:00.722182 657 raft_consensus.cc:2258] T 0ae1063008d44f6eaa61a6e353732e48 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:00.722702 980 raft_consensus.cc:2258] T 8fefbfb003914bf28c4176cd54a9130f P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:00.723381 657 ts_tablet_manager.cc:1875] T 0ae1063008d44f6eaa61a6e353732e48 P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:00.724210 980 ts_tablet_manager.cc:1875] T 8fefbfb003914bf28c4176cd54a9130f P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:00.725481 656 tablet_replica.cc:330] T 0ae1063008d44f6eaa61a6e353732e48 P 1f10f80f13c84e80b03eda7085a3b012: stopping tablet replica
I20240503 02:13:00.725783 656 raft_consensus.cc:2229] T 0ae1063008d44f6eaa61a6e353732e48 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Raft consensus shutting down.
W20240503 02:13:00.726740 30070 consensus_peers.cc:485] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8fdbaf1df10b46d5a47326415d0cd0e9. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:00.727290 656 raft_consensus.cc:2258] T 0ae1063008d44f6eaa61a6e353732e48 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:00.728338 656 ts_tablet_manager.cc:1875] T 0ae1063008d44f6eaa61a6e353732e48 P 1f10f80f13c84e80b03eda7085a3b012: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:13:00.733142 29942 consensus_peers.cc:485] T 8fefbfb003914bf28c4176cd54a9130f P 155735c7f8b14997ac4c770aee2a9401 -> Peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Couldn't send request to peer 16440e365430405fb10366ab70e2b429. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:00.733740 29729 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=24bee9471db248d98a23cbedf87d8255] alter complete (version 2)
I20240503 02:13:00.734911 656 ts_tablet_manager.cc:1888] T 0ae1063008d44f6eaa61a6e353732e48 P 1f10f80f13c84e80b03eda7085a3b012: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:00.735083 656 log.cc:1196] T 0ae1063008d44f6eaa61a6e353732e48 P 1f10f80f13c84e80b03eda7085a3b012: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/wals/0ae1063008d44f6eaa61a6e353732e48
I20240503 02:13:00.735584 656 ts_tablet_manager.cc:1909] T 0ae1063008d44f6eaa61a6e353732e48 P 1f10f80f13c84e80b03eda7085a3b012: Deleting consensus metadata
I20240503 02:13:00.736600 656 tablet_replica.cc:330] T 8fefbfb003914bf28c4176cd54a9130f P 1f10f80f13c84e80b03eda7085a3b012: stopping tablet replica
I20240503 02:13:00.736773 29719 catalog_manager.cc:4830] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): tablet 0ae1063008d44f6eaa61a6e353732e48 (table rebalance_test_table_18 [id=521f00c464174dec80c6db200c35e142]) successfully deleted
I20240503 02:13:00.736842 657 ts_tablet_manager.cc:1888] T 0ae1063008d44f6eaa61a6e353732e48 P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:00.736845 656 raft_consensus.cc:2229] T 8fefbfb003914bf28c4176cd54a9130f P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:00.736955 657 log.cc:1196] T 0ae1063008d44f6eaa61a6e353732e48 P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/0ae1063008d44f6eaa61a6e353732e48
I20240503 02:13:00.737133 656 raft_consensus.cc:2258] T 8fefbfb003914bf28c4176cd54a9130f P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:00.737574 980 ts_tablet_manager.cc:1888] T 8fefbfb003914bf28c4176cd54a9130f P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:13:00.737789 980 log.cc:1196] T 8fefbfb003914bf28c4176cd54a9130f P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/8fefbfb003914bf28c4176cd54a9130f
I20240503 02:13:00.737910 656 ts_tablet_manager.cc:1875] T 8fefbfb003914bf28c4176cd54a9130f P 1f10f80f13c84e80b03eda7085a3b012: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:00.738318 657 ts_tablet_manager.cc:1909] T 0ae1063008d44f6eaa61a6e353732e48 P 155735c7f8b14997ac4c770aee2a9401: Deleting consensus metadata
I20240503 02:13:00.738214 1042 consensus_queue.cc:1035] T 9102ab9545c54e82b642673795cb747a P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:00.738891 980 ts_tablet_manager.cc:1909] T 8fefbfb003914bf28c4176cd54a9130f P 16440e365430405fb10366ab70e2b429: Deleting consensus metadata
I20240503 02:13:00.739939 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet 0ae1063008d44f6eaa61a6e353732e48 (table rebalance_test_table_18 [id=521f00c464174dec80c6db200c35e142]) successfully deleted
I20240503 02:13:00.740964 657 tablet_replica.cc:330] T 8fefbfb003914bf28c4176cd54a9130f P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:13:00.741586 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet 8fefbfb003914bf28c4176cd54a9130f (table rebalance_test_table_18 [id=521f00c464174dec80c6db200c35e142]) successfully deleted
I20240503 02:13:00.741771 1231 tablet.cc:1722] T ab995d6a8aeb42218230cf4f7cda448b P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:00.742030 657 raft_consensus.cc:2229] T 8fefbfb003914bf28c4176cd54a9130f P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:00.742283 1231 tablet.cc:1620] T ab995d6a8aeb42218230cf4f7cda448b P f575ce92c0224eeb80b9fae5f6561fbe: MemRowSet was empty: no flush needed.
I20240503 02:13:00.742336 657 raft_consensus.cc:2258] T 8fefbfb003914bf28c4176cd54a9130f P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:00.744065 657 ts_tablet_manager.cc:1875] T 8fefbfb003914bf28c4176cd54a9130f P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:00.745956 656 ts_tablet_manager.cc:1888] T 8fefbfb003914bf28c4176cd54a9130f P 1f10f80f13c84e80b03eda7085a3b012: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:13:00.746188 656 log.cc:1196] T 8fefbfb003914bf28c4176cd54a9130f P 1f10f80f13c84e80b03eda7085a3b012: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/wals/8fefbfb003914bf28c4176cd54a9130f
I20240503 02:13:00.746827 656 ts_tablet_manager.cc:1909] T 8fefbfb003914bf28c4176cd54a9130f P 1f10f80f13c84e80b03eda7085a3b012: Deleting consensus metadata
I20240503 02:13:00.748170 29719 catalog_manager.cc:4830] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): tablet 8fefbfb003914bf28c4176cd54a9130f (table rebalance_test_table_18 [id=521f00c464174dec80c6db200c35e142]) successfully deleted
I20240503 02:13:00.750694 657 ts_tablet_manager.cc:1888] T 8fefbfb003914bf28c4176cd54a9130f P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:13:00.750869 657 log.cc:1196] T 8fefbfb003914bf28c4176cd54a9130f P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/8fefbfb003914bf28c4176cd54a9130f
I20240503 02:13:00.751338 657 ts_tablet_manager.cc:1909] T 8fefbfb003914bf28c4176cd54a9130f P 155735c7f8b14997ac4c770aee2a9401: Deleting consensus metadata
I20240503 02:13:00.752496 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet 8fefbfb003914bf28c4176cd54a9130f (table rebalance_test_table_18 [id=521f00c464174dec80c6db200c35e142]) successfully deleted
I20240503 02:13:00.765404 436 raft_consensus.cc:3037] T 9102ab9545c54e82b642673795cb747a P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:00.769894 702 consensus_queue.cc:1035] T 9102ab9545c54e82b642673795cb747a P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:00.788709 30264 consensus_queue.cc:237] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:00.791471 30134 raft_consensus.cc:1261] T 7527cc5931e24fa180efedbf81438efd P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:00.791471 29874 raft_consensus.cc:1261] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:00.792100 1180 consensus_queue.cc:1035] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:00.792410 1179 consensus_queue.cc:1035] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:13:00.797857 30199 consensus_peers.cc:485] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7527cc5931e24fa180efedbf81438efd. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:00.797763 1180 raft_consensus.cc:2932] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:00.798509 30134 raft_consensus.cc:2932] T 7527cc5931e24fa180efedbf81438efd P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:00.806792 29731 catalog_manager.cc:5484] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: config changed from index -1 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:00.807255 29874 raft_consensus.cc:2932] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:00.818876 1200 consensus_queue.cc:1035] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:00.837790 1181 consensus_queue.cc:1035] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:00.840759 29731 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:13:00.853374 1227 tablet.cc:1722] T ab995d6a8aeb42218230cf4f7cda448b P 155735c7f8b14997ac4c770aee2a9401: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:00.854776 1228 tablet.cc:1722] T ab995d6a8aeb42218230cf4f7cda448b P 9a4d92bc25374fb3960fde7d10be309d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:00.863026 29729 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table_new_name [id=24bee9471db248d98a23cbedf87d8255] alter complete (version 3)
I20240503 02:13:00.866744 1231 tablet.cc:1722] T ab995d6a8aeb42218230cf4f7cda448b P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:00.956710 30006 consensus_queue.cc:237] T 08ee37b160d441ddabb640be907d6771 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:00.959295 30264 raft_consensus.cc:1261] T 08ee37b160d441ddabb640be907d6771 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:00.959347 30134 raft_consensus.cc:1261] T 08ee37b160d441ddabb640be907d6771 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:13:00.959666 29940 consensus_peers.cc:485] T 08ee37b160d441ddabb640be907d6771 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): Couldn't send request to peer 8a666e798fe8477d8dd1e51b385d5ead. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 08ee37b160d441ddabb640be907d6771. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:00.959889 986 consensus_queue.cc:1035] T 08ee37b160d441ddabb640be907d6771 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:00.960801 986 consensus_queue.cc:1035] T 08ee37b160d441ddabb640be907d6771 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:00.963268 778 raft_consensus.cc:2932] T 08ee37b160d441ddabb640be907d6771 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:00.963943 30264 raft_consensus.cc:2932] T 08ee37b160d441ddabb640be907d6771 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:00.967741 30136 raft_consensus.cc:2932] T 08ee37b160d441ddabb640be907d6771 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:00.968935 29731 catalog_manager.cc:5484] T 08ee37b160d441ddabb640be907d6771 P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: config changed from index -1 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:00.980370 29734 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:00.980618 29734 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:00.984402 29734 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet ab995d6a8aeb42218230cf4f7cda448b
I20240503 02:13:00.985357 32620 tablet_service.cc:1514] Processing DeleteTablet for tablet ab995d6a8aeb42218230cf4f7cda448b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:00 UTC) from {username='slave'} at 127.0.0.1:38200
I20240503 02:13:00.985374 29986 tablet_service.cc:1514] Processing DeleteTablet for tablet ab995d6a8aeb42218230cf4f7cda448b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:00 UTC) from {username='slave'} at 127.0.0.1:52158
I20240503 02:13:00.985647 657 tablet_replica.cc:330] T ab995d6a8aeb42218230cf4f7cda448b P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:13:00.985723 1271 tablet_replica.cc:330] T ab995d6a8aeb42218230cf4f7cda448b P 9a4d92bc25374fb3960fde7d10be309d: stopping tablet replica
I20240503 02:13:00.985857 657 raft_consensus.cc:2229] T ab995d6a8aeb42218230cf4f7cda448b P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:00.985947 1271 raft_consensus.cc:2229] T ab995d6a8aeb42218230cf4f7cda448b P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:00.986124 657 raft_consensus.cc:2258] T ab995d6a8aeb42218230cf4f7cda448b P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:00.986191 1271 raft_consensus.cc:2258] T ab995d6a8aeb42218230cf4f7cda448b P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:00.986433 546 tablet_service.cc:1514] Processing DeleteTablet for tablet ab995d6a8aeb42218230cf4f7cda448b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:00 UTC) from {username='slave'} at 127.0.0.1:39028
I20240503 02:13:00.987058 657 ts_tablet_manager.cc:1875] T ab995d6a8aeb42218230cf4f7cda448b P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:00.988816 1326 ts_tablet_manager.cc:902] T 83c7d1060ccf49a2adda4302600a9876 P 1f10f80f13c84e80b03eda7085a3b012: Initiating tablet copy from peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:00.991034 1271 ts_tablet_manager.cc:1875] T ab995d6a8aeb42218230cf4f7cda448b P 9a4d92bc25374fb3960fde7d10be309d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:00.995004 657 ts_tablet_manager.cc:1888] T ab995d6a8aeb42218230cf4f7cda448b P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:00.995245 657 log.cc:1196] T ab995d6a8aeb42218230cf4f7cda448b P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/ab995d6a8aeb42218230cf4f7cda448b
I20240503 02:13:00.995793 657 ts_tablet_manager.cc:1909] T ab995d6a8aeb42218230cf4f7cda448b P 155735c7f8b14997ac4c770aee2a9401: Deleting consensus metadata
I20240503 02:13:00.997007 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet ab995d6a8aeb42218230cf4f7cda448b (table rebalancer_dynamic_table_new_name [id=24bee9471db248d98a23cbedf87d8255]) successfully deleted
I20240503 02:13:01.005060 1325 tablet_replica.cc:330] T ab995d6a8aeb42218230cf4f7cda448b P f575ce92c0224eeb80b9fae5f6561fbe: stopping tablet replica
I20240503 02:13:01.005592 1325 raft_consensus.cc:2229] T ab995d6a8aeb42218230cf4f7cda448b P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:01.006242 1271 ts_tablet_manager.cc:1888] T ab995d6a8aeb42218230cf4f7cda448b P 9a4d92bc25374fb3960fde7d10be309d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:01.006260 1325 raft_consensus.cc:2258] T ab995d6a8aeb42218230cf4f7cda448b P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:01.006537 1271 log.cc:1196] T ab995d6a8aeb42218230cf4f7cda448b P 9a4d92bc25374fb3960fde7d10be309d: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal/wals/ab995d6a8aeb42218230cf4f7cda448b
I20240503 02:13:01.007177 1271 ts_tablet_manager.cc:1909] T ab995d6a8aeb42218230cf4f7cda448b P 9a4d92bc25374fb3960fde7d10be309d: Deleting consensus metadata
I20240503 02:13:01.007716 1325 ts_tablet_manager.cc:1875] T ab995d6a8aeb42218230cf4f7cda448b P f575ce92c0224eeb80b9fae5f6561fbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:01.008627 29719 catalog_manager.cc:4830] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): tablet ab995d6a8aeb42218230cf4f7cda448b (table rebalancer_dynamic_table_new_name [id=24bee9471db248d98a23cbedf87d8255]) successfully deleted
I20240503 02:13:01.012760 1326 tablet_copy_client.cc:308] T 83c7d1060ccf49a2adda4302600a9876 P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.7:44241
I20240503 02:13:01.014106 457 tablet_copy_service.cc:140] P fbbc2be6382e47908ecb7c68c367bd26: Received BeginTabletCopySession request for tablet 83c7d1060ccf49a2adda4302600a9876 from peer 1f10f80f13c84e80b03eda7085a3b012 ({username='slave'} at 127.22.108.4:42187)
I20240503 02:13:01.014364 457 tablet_copy_service.cc:161] P fbbc2be6382e47908ecb7c68c367bd26: Beginning new tablet copy session on tablet 83c7d1060ccf49a2adda4302600a9876 from peer 1f10f80f13c84e80b03eda7085a3b012 at {username='slave'} at 127.22.108.4:42187: session id = 1f10f80f13c84e80b03eda7085a3b012-83c7d1060ccf49a2adda4302600a9876
I20240503 02:13:01.018075 457 tablet_copy_source_session.cc:205] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:01.018935 1325 ts_tablet_manager.cc:1888] T ab995d6a8aeb42218230cf4f7cda448b P f575ce92c0224eeb80b9fae5f6561fbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:01.019160 1325 log.cc:1196] T ab995d6a8aeb42218230cf4f7cda448b P f575ce92c0224eeb80b9fae5f6561fbe: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal/wals/ab995d6a8aeb42218230cf4f7cda448b
I20240503 02:13:01.019739 1325 ts_tablet_manager.cc:1909] T ab995d6a8aeb42218230cf4f7cda448b P f575ce92c0224eeb80b9fae5f6561fbe: Deleting consensus metadata
I20240503 02:13:01.021011 1326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83c7d1060ccf49a2adda4302600a9876. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.021123 29718 catalog_manager.cc:4830] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): tablet ab995d6a8aeb42218230cf4f7cda448b (table rebalancer_dynamic_table_new_name [id=24bee9471db248d98a23cbedf87d8255]) successfully deleted
I20240503 02:13:01.029917 1326 tablet_copy_client.cc:783] T 83c7d1060ccf49a2adda4302600a9876 P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:01.030300 1326 tablet_copy_client.cc:647] T 83c7d1060ccf49a2adda4302600a9876 P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:01.035490 1326 tablet_copy_client.cc:523] T 83c7d1060ccf49a2adda4302600a9876 P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:01.039943 1326 tablet_bootstrap.cc:492] T 83c7d1060ccf49a2adda4302600a9876 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:13:01.055080 1326 tablet_bootstrap.cc:492] T 83c7d1060ccf49a2adda4302600a9876 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:01.055748 1326 tablet_bootstrap.cc:492] T 83c7d1060ccf49a2adda4302600a9876 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap complete.
I20240503 02:13:01.056155 1326 ts_tablet_manager.cc:1367] T 83c7d1060ccf49a2adda4302600a9876 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.016s user 0.016s sys 0.003s
I20240503 02:13:01.056525 30136 consensus_queue.cc:237] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:01.057224 1326 raft_consensus.cc:348] T 83c7d1060ccf49a2adda4302600a9876 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:01.057485 1326 raft_consensus.cc:729] T 83c7d1060ccf49a2adda4302600a9876 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.057792 1326 consensus_queue.cc:260] T 83c7d1060ccf49a2adda4302600a9876 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:01.058559 1326 ts_tablet_manager.cc:1398] T 83c7d1060ccf49a2adda4302600a9876 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:01.059543 457 tablet_copy_service.cc:342] P fbbc2be6382e47908ecb7c68c367bd26: Request end of tablet copy session 1f10f80f13c84e80b03eda7085a3b012-83c7d1060ccf49a2adda4302600a9876 received from {username='slave'} at 127.22.108.4:42187
I20240503 02:13:01.059734 457 tablet_copy_service.cc:433] P fbbc2be6382e47908ecb7c68c367bd26: ending tablet copy session 1f10f80f13c84e80b03eda7085a3b012-83c7d1060ccf49a2adda4302600a9876 on tablet 83c7d1060ccf49a2adda4302600a9876 with peer 1f10f80f13c84e80b03eda7085a3b012
W20240503 02:13:01.059754 30070 consensus_peers.cc:485] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7076453cbe5944eb9b075127e1e0495d. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:01.060120 29872 raft_consensus.cc:1261] T 7076453cbe5944eb9b075127e1e0495d P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:01.060674 30006 raft_consensus.cc:1261] T 7076453cbe5944eb9b075127e1e0495d P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:01.060670 912 consensus_queue.cc:1035] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:01.062233 912 consensus_queue.cc:1035] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:01.066648 912 raft_consensus.cc:2932] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:01.067044 30006 raft_consensus.cc:2932] T 7076453cbe5944eb9b075127e1e0495d P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:01.069952 29734 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_extra_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:01.072602 29731 catalog_manager.cc:5484] T 7076453cbe5944eb9b075127e1e0495d P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: config changed from index -1 to 4, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:01.075690 29872 raft_consensus.cc:2932] T 7076453cbe5944eb9b075127e1e0495d P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:01.084995 29986 tablet_service.cc:1467] Processing CreateTablet for tablet a52bee1fdc9d48b1a3463ecd62c8b44e (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=81455612cf1243e09bae574807b9303b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:01.085479 29986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a52bee1fdc9d48b1a3463ecd62c8b44e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.091627 1181 consensus_queue.cc:1035] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:01.092415 32754 tablet_service.cc:1467] Processing CreateTablet for tablet a52bee1fdc9d48b1a3463ecd62c8b44e (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=81455612cf1243e09bae574807b9303b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:01.092907 32754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a52bee1fdc9d48b1a3463ecd62c8b44e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.093187 546 tablet_service.cc:1467] Processing CreateTablet for tablet 3c437d9d466348f2863c407666c348aa (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=81455612cf1243e09bae574807b9303b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:01.094014 546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c437d9d466348f2863c407666c348aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.094255 547 tablet_service.cc:1467] Processing CreateTablet for tablet 6bd4fcc4faf44d148a03beaafc1c05ba (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=81455612cf1243e09bae574807b9303b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:01.094652 547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bd4fcc4faf44d148a03beaafc1c05ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.095541 415 tablet_service.cc:1467] Processing CreateTablet for tablet a52bee1fdc9d48b1a3463ecd62c8b44e (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=81455612cf1243e09bae574807b9303b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:01.095954 415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a52bee1fdc9d48b1a3463ecd62c8b44e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.098101 29729 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:13:01.101420 32753 tablet_service.cc:1467] Processing CreateTablet for tablet 3c437d9d466348f2863c407666c348aa (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=81455612cf1243e09bae574807b9303b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:01.101838 32753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c437d9d466348f2863c407666c348aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.104233 417 tablet_service.cc:1467] Processing CreateTablet for tablet 3c437d9d466348f2863c407666c348aa (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=81455612cf1243e09bae574807b9303b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:01.104631 417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c437d9d466348f2863c407666c348aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.111690 29853 tablet_service.cc:1467] Processing CreateTablet for tablet 6bd4fcc4faf44d148a03beaafc1c05ba (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=81455612cf1243e09bae574807b9303b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:01.112147 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bd4fcc4faf44d148a03beaafc1c05ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.113713 762 consensus_queue.cc:1035] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:01.120767 416 tablet_service.cc:1467] Processing CreateTablet for tablet 3b840a1964d049f8a9948a09c059a068 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c985f9c1d7a040d793cf0eaab4c1c6c5]), partition=
I20240503 02:13:01.120841 29984 tablet_service.cc:1467] Processing CreateTablet for tablet 3b840a1964d049f8a9948a09c059a068 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c985f9c1d7a040d793cf0eaab4c1c6c5]), partition=
I20240503 02:13:01.121161 416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b840a1964d049f8a9948a09c059a068. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.121201 29984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b840a1964d049f8a9948a09c059a068. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.124855 32755 tablet_service.cc:1467] Processing CreateTablet for tablet 6bd4fcc4faf44d148a03beaafc1c05ba (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=81455612cf1243e09bae574807b9303b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:01.125321 32755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bd4fcc4faf44d148a03beaafc1c05ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.129395 32752 tablet_service.cc:1467] Processing CreateTablet for tablet 3b840a1964d049f8a9948a09c059a068 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c985f9c1d7a040d793cf0eaab4c1c6c5]), partition=
I20240503 02:13:01.129717 32752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b840a1964d049f8a9948a09c059a068. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.141526 1353 tablet_bootstrap.cc:492] T 3b840a1964d049f8a9948a09c059a068 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:01.143440 1353 tablet_bootstrap.cc:654] T 3b840a1964d049f8a9948a09c059a068 P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:01.153937 1353 tablet_bootstrap.cc:492] T 3b840a1964d049f8a9948a09c059a068 P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:01.154771 1353 ts_tablet_manager.cc:1367] T 3b840a1964d049f8a9948a09c059a068 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20240503 02:13:01.155913 1353 raft_consensus.cc:348] T 3b840a1964d049f8a9948a09c059a068 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:01.157498 1353 raft_consensus.cc:374] T 3b840a1964d049f8a9948a09c059a068 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:01.157781 1353 raft_consensus.cc:729] T 3b840a1964d049f8a9948a09c059a068 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.158337 1353 consensus_queue.cc:260] T 3b840a1964d049f8a9948a09c059a068 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:01.159085 1357 tablet_bootstrap.cc:492] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:13:01.160085 1350 tablet_bootstrap.cc:492] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:01.162063 1357 tablet_bootstrap.cc:654] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:01.162356 1353 ts_tablet_manager.cc:1398] T 3b840a1964d049f8a9948a09c059a068 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:13:01.162379 1350 tablet_bootstrap.cc:654] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:01.162772 1356 tablet_bootstrap.cc:492] T 6bd4fcc4faf44d148a03beaafc1c05ba P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:01.165975 1349 tablet_bootstrap.cc:492] T a52bee1fdc9d48b1a3463ecd62c8b44e P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:13:01.166179 1353 tablet_bootstrap.cc:492] T 3c437d9d466348f2863c407666c348aa P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:01.167789 1349 tablet_bootstrap.cc:654] T a52bee1fdc9d48b1a3463ecd62c8b44e P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:01.168234 1356 tablet_bootstrap.cc:654] T 6bd4fcc4faf44d148a03beaafc1c05ba P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:01.169677 1353 tablet_bootstrap.cc:654] T 3c437d9d466348f2863c407666c348aa P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:01.170567 1350 tablet_bootstrap.cc:492] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:01.170822 1350 ts_tablet_manager.cc:1367] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.004s
I20240503 02:13:01.171865 1350 raft_consensus.cc:348] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:01.172252 1350 raft_consensus.cc:374] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:01.172408 1350 raft_consensus.cc:729] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.172724 1350 consensus_queue.cc:260] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:01.174114 1349 tablet_bootstrap.cc:492] T a52bee1fdc9d48b1a3463ecd62c8b44e P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:13:01.174350 1349 ts_tablet_manager.cc:1367] T a52bee1fdc9d48b1a3463ecd62c8b44e P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240503 02:13:01.175330 1349 raft_consensus.cc:348] T a52bee1fdc9d48b1a3463ecd62c8b44e P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:01.176193 1349 raft_consensus.cc:374] T a52bee1fdc9d48b1a3463ecd62c8b44e P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:01.176357 1349 raft_consensus.cc:729] T a52bee1fdc9d48b1a3463ecd62c8b44e P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.176522 1350 ts_tablet_manager.cc:1398] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:13:01.176749 1349 consensus_queue.cc:260] T a52bee1fdc9d48b1a3463ecd62c8b44e P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:01.177464 1348 ts_tablet_manager.cc:902] T 5f4b72b138c64ac58e6477cd2b1fb215 P 8a666e798fe8477d8dd1e51b385d5ead: Initiating tablet copy from peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:13:01.178752 1350 tablet_bootstrap.cc:492] T 3c437d9d466348f2863c407666c348aa P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:01.178767 1349 ts_tablet_manager.cc:1398] T a52bee1fdc9d48b1a3463ecd62c8b44e P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:13:01.179507 1349 tablet_bootstrap.cc:492] T 3b840a1964d049f8a9948a09c059a068 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:13:01.179589 1113 raft_consensus.cc:482] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:01.178277 1356 tablet_bootstrap.cc:492] T 6bd4fcc4faf44d148a03beaafc1c05ba P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:01.179752 1113 raft_consensus.cc:504] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:01.179970 1356 ts_tablet_manager.cc:1367] T 6bd4fcc4faf44d148a03beaafc1c05ba P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.006s
I20240503 02:13:01.180466 1113 leader_election.cc:288] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:13:01.180800 1356 raft_consensus.cc:348] T 6bd4fcc4faf44d148a03beaafc1c05ba P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:01.181003 1356 raft_consensus.cc:374] T 6bd4fcc4faf44d148a03beaafc1c05ba P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:01.181128 1356 raft_consensus.cc:729] T 6bd4fcc4faf44d148a03beaafc1c05ba P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.181239 1349 tablet_bootstrap.cc:654] T 3b840a1964d049f8a9948a09c059a068 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:01.181356 29872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bd4fcc4faf44d148a03beaafc1c05ba" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:13:01.181360 1356 consensus_queue.cc:260] T 6bd4fcc4faf44d148a03beaafc1c05ba P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:01.181953 1356 ts_tablet_manager.cc:1398] T 6bd4fcc4faf44d148a03beaafc1c05ba P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:13:01.182723 1356 tablet_bootstrap.cc:492] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
W20240503 02:13:01.182741 504 leader_election.cc:341] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:01.183271 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bd4fcc4faf44d148a03beaafc1c05ba" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:13:01.183475 305 raft_consensus.cc:2454] T 6bd4fcc4faf44d148a03beaafc1c05ba P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 0.
I20240503 02:13:01.183933 502 leader_election.cc:302] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, f575ce92c0224eeb80b9fae5f6561fbe; no voters: 16440e365430405fb10366ab70e2b429
I20240503 02:13:01.184367 1113 raft_consensus.cc:2781] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:01.184482 1113 raft_consensus.cc:482] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:01.184576 1113 raft_consensus.cc:3037] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.185272 1356 tablet_bootstrap.cc:654] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:01.187567 1113 raft_consensus.cc:504] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:01.188629 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bd4fcc4faf44d148a03beaafc1c05ba" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:13:01.188849 307 raft_consensus.cc:3037] T 6bd4fcc4faf44d148a03beaafc1c05ba P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.189008 1353 tablet_bootstrap.cc:492] T 3c437d9d466348f2863c407666c348aa P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:01.188987 29872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bd4fcc4faf44d148a03beaafc1c05ba" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:13:01.189224 1353 ts_tablet_manager.cc:1367] T 3c437d9d466348f2863c407666c348aa P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.002s
W20240503 02:13:01.189553 504 leader_election.cc:341] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:01.189757 1113 leader_election.cc:288] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 election: Requested vote from peers 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:13:01.191437 1350 tablet_bootstrap.cc:654] T 3c437d9d466348f2863c407666c348aa P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:01.191474 307 raft_consensus.cc:2454] T 6bd4fcc4faf44d148a03beaafc1c05ba P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 1.
I20240503 02:13:01.191893 1353 raft_consensus.cc:348] T 3c437d9d466348f2863c407666c348aa P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:01.192212 502 leader_election.cc:302] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, f575ce92c0224eeb80b9fae5f6561fbe; no voters: 16440e365430405fb10366ab70e2b429
I20240503 02:13:01.192716 1113 raft_consensus.cc:2781] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:01.193166 1349 tablet_bootstrap.cc:492] T 3b840a1964d049f8a9948a09c059a068 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:13:01.193678 1349 ts_tablet_manager.cc:1367] T 3b840a1964d049f8a9948a09c059a068 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.004s
I20240503 02:13:01.198464 1349 raft_consensus.cc:348] T 3b840a1964d049f8a9948a09c059a068 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:01.199002 1353 raft_consensus.cc:374] T 3c437d9d466348f2863c407666c348aa P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:01.199360 1353 raft_consensus.cc:729] T 3c437d9d466348f2863c407666c348aa P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.200541 1353 consensus_queue.cc:260] T 3c437d9d466348f2863c407666c348aa P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:01.201835 1113 raft_consensus.cc:686] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe [term 1 LEADER]: Becoming Leader. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Running, Role: LEADER
I20240503 02:13:01.202297 1113 consensus_queue.cc:237] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:01.203500 1353 ts_tablet_manager.cc:1398] T 3c437d9d466348f2863c407666c348aa P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20240503 02:13:01.203684 1349 raft_consensus.cc:374] T 3b840a1964d049f8a9948a09c059a068 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:01.203876 1349 raft_consensus.cc:729] T 3b840a1964d049f8a9948a09c059a068 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.205173 1353 tablet_bootstrap.cc:492] T a52bee1fdc9d48b1a3463ecd62c8b44e P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:01.205415 1349 consensus_queue.cc:260] T 3b840a1964d049f8a9948a09c059a068 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:01.207232 1353 tablet_bootstrap.cc:654] T a52bee1fdc9d48b1a3463ecd62c8b44e P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:01.207260 1350 tablet_bootstrap.cc:492] T 3c437d9d466348f2863c407666c348aa P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:01.207805 1350 ts_tablet_manager.cc:1367] T 3c437d9d466348f2863c407666c348aa P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.003s
I20240503 02:13:01.209102 1350 raft_consensus.cc:348] T 3c437d9d466348f2863c407666c348aa P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:01.209295 1349 ts_tablet_manager.cc:1398] T 3b840a1964d049f8a9948a09c059a068 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.015s user 0.000s sys 0.002s
I20240503 02:13:01.209811 1350 raft_consensus.cc:374] T 3c437d9d466348f2863c407666c348aa P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:01.209918 1350 raft_consensus.cc:729] T 3c437d9d466348f2863c407666c348aa P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.210505 1350 consensus_queue.cc:260] T 3c437d9d466348f2863c407666c348aa P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:01.214232 1350 ts_tablet_manager.cc:1398] T 3c437d9d466348f2863c407666c348aa P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:13:01.217967 1353 tablet_bootstrap.cc:492] T a52bee1fdc9d48b1a3463ecd62c8b44e P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:01.218202 1353 ts_tablet_manager.cc:1367] T a52bee1fdc9d48b1a3463ecd62c8b44e P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.007s
I20240503 02:13:01.218973 29725 catalog_manager.cc:5484] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: term changed from 0 to 1, leader changed from <none> to f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8). New cstate: current_term: 1 leader_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:01.219235 1353 raft_consensus.cc:348] T a52bee1fdc9d48b1a3463ecd62c8b44e P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:01.219643 1353 raft_consensus.cc:374] T a52bee1fdc9d48b1a3463ecd62c8b44e P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:01.219805 1353 raft_consensus.cc:729] T a52bee1fdc9d48b1a3463ecd62c8b44e P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.220124 1353 consensus_queue.cc:260] T a52bee1fdc9d48b1a3463ecd62c8b44e P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:01.224406 1353 ts_tablet_manager.cc:1398] T a52bee1fdc9d48b1a3463ecd62c8b44e P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 02:13:01.225322 1376 ts_tablet_manager.cc:902] T 96de1c26ae884ea18b63b328db040ba1 P fbbc2be6382e47908ecb7c68c367bd26: Initiating tablet copy from peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:13:01.227317 1380 ts_tablet_manager.cc:902] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 9a4d92bc25374fb3960fde7d10be309d: Initiating tablet copy from peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:13:01.231783 1356 tablet_bootstrap.cc:492] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:01.232827 1356 ts_tablet_manager.cc:1367] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.050s user 0.004s sys 0.008s
I20240503 02:13:01.234134 1356 raft_consensus.cc:348] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:01.234586 1356 raft_consensus.cc:374] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:01.234740 1356 raft_consensus.cc:729] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.235168 1356 consensus_queue.cc:260] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:01.237119 1356 ts_tablet_manager.cc:1398] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:13:01.237722 1356 tablet_bootstrap.cc:492] T a52bee1fdc9d48b1a3463ecd62c8b44e P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:01.244716 1356 tablet_bootstrap.cc:654] T a52bee1fdc9d48b1a3463ecd62c8b44e P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:01.253957 1357 tablet_bootstrap.cc:492] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:13:01.254288 1357 ts_tablet_manager.cc:1367] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.096s user 0.004s sys 0.009s
I20240503 02:13:01.254925 1356 tablet_bootstrap.cc:492] T a52bee1fdc9d48b1a3463ecd62c8b44e P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:01.255373 1357 raft_consensus.cc:348] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:01.255702 1357 raft_consensus.cc:374] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:01.255860 1357 raft_consensus.cc:729] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.256145 1356 ts_tablet_manager.cc:1367] T a52bee1fdc9d48b1a3463ecd62c8b44e P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.003s
I20240503 02:13:01.256247 1357 consensus_queue.cc:260] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:01.257529 1356 raft_consensus.cc:348] T a52bee1fdc9d48b1a3463ecd62c8b44e P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:01.258677 1357 ts_tablet_manager.cc:1398] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:13:01.259224 1356 raft_consensus.cc:374] T a52bee1fdc9d48b1a3463ecd62c8b44e P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:01.259536 1356 raft_consensus.cc:729] T a52bee1fdc9d48b1a3463ecd62c8b44e P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.260080 1356 consensus_queue.cc:260] T a52bee1fdc9d48b1a3463ecd62c8b44e P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:01.262192 1356 ts_tablet_manager.cc:1398] T a52bee1fdc9d48b1a3463ecd62c8b44e P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:13:01.263077 1356 tablet_bootstrap.cc:492] T 3b840a1964d049f8a9948a09c059a068 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:01.267525 1380 tablet_copy_client.cc:308] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.3:42679
I20240503 02:13:01.268736 1356 tablet_bootstrap.cc:654] T 3b840a1964d049f8a9948a09c059a068 P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:01.281296 1376 tablet_copy_client.cc:308] T 96de1c26ae884ea18b63b328db040ba1 P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.4:33935
I20240503 02:13:01.282289 30286 tablet_copy_service.cc:140] P 1f10f80f13c84e80b03eda7085a3b012: Received BeginTabletCopySession request for tablet 96de1c26ae884ea18b63b328db040ba1 from peer fbbc2be6382e47908ecb7c68c367bd26 ({username='slave'} at 127.22.108.7:52333)
I20240503 02:13:01.282509 30286 tablet_copy_service.cc:161] P 1f10f80f13c84e80b03eda7085a3b012: Beginning new tablet copy session on tablet 96de1c26ae884ea18b63b328db040ba1 from peer fbbc2be6382e47908ecb7c68c367bd26 at {username='slave'} at 127.22.108.7:52333: session id = fbbc2be6382e47908ecb7c68c367bd26-96de1c26ae884ea18b63b328db040ba1
I20240503 02:13:01.286048 30286 tablet_copy_source_session.cc:205] T 96de1c26ae884ea18b63b328db040ba1 P 1f10f80f13c84e80b03eda7085a3b012: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:01.287919 1376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96de1c26ae884ea18b63b328db040ba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.288750 30156 tablet_copy_service.cc:140] P a9094240860e43bfb129193f3bfd5be6: Received BeginTabletCopySession request for tablet 8fdbaf1df10b46d5a47326415d0cd0e9 from peer 9a4d92bc25374fb3960fde7d10be309d ({username='slave'} at 127.22.108.5:33825)
I20240503 02:13:01.288956 30156 tablet_copy_service.cc:161] P a9094240860e43bfb129193f3bfd5be6: Beginning new tablet copy session on tablet 8fdbaf1df10b46d5a47326415d0cd0e9 from peer 9a4d92bc25374fb3960fde7d10be309d at {username='slave'} at 127.22.108.5:33825: session id = 9a4d92bc25374fb3960fde7d10be309d-8fdbaf1df10b46d5a47326415d0cd0e9
I20240503 02:13:01.292716 30156 tablet_copy_source_session.cc:205] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:01.294330 1376 tablet_copy_client.cc:783] T 96de1c26ae884ea18b63b328db040ba1 P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:01.294628 1376 tablet_copy_client.cc:647] T 96de1c26ae884ea18b63b328db040ba1 P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:01.295207 1380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fdbaf1df10b46d5a47326415d0cd0e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.298097 1376 tablet_copy_client.cc:523] T 96de1c26ae884ea18b63b328db040ba1 P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:01.299799 762 raft_consensus.cc:482] T 3b840a1964d049f8a9948a09c059a068 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:01.299962 762 raft_consensus.cc:504] T 3b840a1964d049f8a9948a09c059a068 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:01.300660 762 leader_election.cc:288] T 3b840a1964d049f8a9948a09c059a068 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:13:01.301189 30006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b840a1964d049f8a9948a09c059a068" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:13:01.301206 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b840a1964d049f8a9948a09c059a068" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:13:01.301343 1376 tablet_bootstrap.cc:492] T 96de1c26ae884ea18b63b328db040ba1 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:01.301429 30006 raft_consensus.cc:2454] T 3b840a1964d049f8a9948a09c059a068 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 0.
W20240503 02:13:01.301802 371 leader_election.cc:341] T 3b840a1964d049f8a9948a09c059a068 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:01.301889 1377 ts_tablet_manager.cc:902] T 7527cc5931e24fa180efedbf81438efd P f575ce92c0224eeb80b9fae5f6561fbe: Initiating tablet copy from peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:13:01.303210 371 leader_election.cc:302] T 3b840a1964d049f8a9948a09c059a068 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, fbbc2be6382e47908ecb7c68c367bd26; no voters: 8a666e798fe8477d8dd1e51b385d5ead
I20240503 02:13:01.303745 762 raft_consensus.cc:2781] T 3b840a1964d049f8a9948a09c059a068 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:01.303874 762 raft_consensus.cc:482] T 3b840a1964d049f8a9948a09c059a068 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:01.303972 762 raft_consensus.cc:3037] T 3b840a1964d049f8a9948a09c059a068 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.305346 1348 tablet_copy_client.cc:308] T 5f4b72b138c64ac58e6477cd2b1fb215 P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.1:36719
I20240503 02:13:01.306551 29895 tablet_copy_service.cc:140] P 16440e365430405fb10366ab70e2b429: Received BeginTabletCopySession request for tablet 5f4b72b138c64ac58e6477cd2b1fb215 from peer 8a666e798fe8477d8dd1e51b385d5ead ({username='slave'} at 127.22.108.6:49099)
I20240503 02:13:01.306766 29895 tablet_copy_service.cc:161] P 16440e365430405fb10366ab70e2b429: Beginning new tablet copy session on tablet 5f4b72b138c64ac58e6477cd2b1fb215 from peer 8a666e798fe8477d8dd1e51b385d5ead at {username='slave'} at 127.22.108.6:49099: session id = 8a666e798fe8477d8dd1e51b385d5ead-5f4b72b138c64ac58e6477cd2b1fb215
I20240503 02:13:01.306802 762 raft_consensus.cc:504] T 3b840a1964d049f8a9948a09c059a068 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:01.307461 762 leader_election.cc:288] T 3b840a1964d049f8a9948a09c059a068 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Requested vote from peers 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:13:01.308413 1356 tablet_bootstrap.cc:492] T 3b840a1964d049f8a9948a09c059a068 P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:01.308591 1356 ts_tablet_manager.cc:1367] T 3b840a1964d049f8a9948a09c059a068 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.002s
I20240503 02:13:01.308775 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b840a1964d049f8a9948a09c059a068" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:13:01.309316 1356 raft_consensus.cc:348] T 3b840a1964d049f8a9948a09c059a068 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:01.309549 1356 raft_consensus.cc:374] T 3b840a1964d049f8a9948a09c059a068 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:01.309631 1356 raft_consensus.cc:729] T 3b840a1964d049f8a9948a09c059a068 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.309681 30006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b840a1964d049f8a9948a09c059a068" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:13:01.309923 30006 raft_consensus.cc:3037] T 3b840a1964d049f8a9948a09c059a068 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.309883 1356 consensus_queue.cc:260] T 3b840a1964d049f8a9948a09c059a068 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:01.310357 1356 ts_tablet_manager.cc:1398] T 3b840a1964d049f8a9948a09c059a068 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:01.310609 29895 tablet_copy_source_session.cc:205] T 5f4b72b138c64ac58e6477cd2b1fb215 P 16440e365430405fb10366ab70e2b429: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:01.313282 1348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f4b72b138c64ac58e6477cd2b1fb215. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:13:01.322039 371 leader_election.cc:341] T 3b840a1964d049f8a9948a09c059a068 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:01.322522 1348 tablet_copy_client.cc:783] T 5f4b72b138c64ac58e6477cd2b1fb215 P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:01.322885 1348 tablet_copy_client.cc:647] T 5f4b72b138c64ac58e6477cd2b1fb215 P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:01.327358 1348 tablet_copy_client.cc:523] T 5f4b72b138c64ac58e6477cd2b1fb215 P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:01.312651 30006 raft_consensus.cc:2454] T 3b840a1964d049f8a9948a09c059a068 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 1.
I20240503 02:13:01.329500 371 leader_election.cc:302] T 3b840a1964d049f8a9948a09c059a068 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, fbbc2be6382e47908ecb7c68c367bd26; no voters: 8a666e798fe8477d8dd1e51b385d5ead
I20240503 02:13:01.329859 762 raft_consensus.cc:2781] T 3b840a1964d049f8a9948a09c059a068 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:01.330035 762 raft_consensus.cc:686] T 3b840a1964d049f8a9948a09c059a068 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Becoming Leader. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Running, Role: LEADER
I20240503 02:13:01.330396 762 consensus_queue.cc:237] T 3b840a1964d049f8a9948a09c059a068 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:01.331845 1348 tablet_bootstrap.cc:492] T 5f4b72b138c64ac58e6477cd2b1fb215 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:01.335594 29729 catalog_manager.cc:5484] T 3b840a1964d049f8a9948a09c059a068 P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7). New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:01.340971 1377 tablet_copy_client.cc:308] T 7527cc5931e24fa180efedbf81438efd P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.4:33935
I20240503 02:13:01.341811 1380 tablet_copy_client.cc:783] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:01.342231 1380 tablet_copy_client.cc:647] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:01.342401 1183 raft_consensus.cc:482] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:01.342566 30286 tablet_copy_service.cc:140] P 1f10f80f13c84e80b03eda7085a3b012: Received BeginTabletCopySession request for tablet 7527cc5931e24fa180efedbf81438efd from peer f575ce92c0224eeb80b9fae5f6561fbe ({username='slave'} at 127.22.108.8:51621)
I20240503 02:13:01.342605 1183 raft_consensus.cc:504] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:01.342761 30286 tablet_copy_service.cc:161] P 1f10f80f13c84e80b03eda7085a3b012: Beginning new tablet copy session on tablet 7527cc5931e24fa180efedbf81438efd from peer f575ce92c0224eeb80b9fae5f6561fbe at {username='slave'} at 127.22.108.8:51621: session id = f575ce92c0224eeb80b9fae5f6561fbe-7527cc5931e24fa180efedbf81438efd
I20240503 02:13:01.343442 1183 leader_election.cc:288] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:13:01.344342 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c437d9d466348f2863c407666c348aa" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:13:01.344588 436 raft_consensus.cc:2454] T 3c437d9d466348f2863c407666c348aa P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:13:01.345197 32709 leader_election.cc:302] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:01.345782 30286 tablet_copy_source_session.cc:205] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:01.347905 1183 raft_consensus.cc:2781] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:01.348027 1183 raft_consensus.cc:482] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:01.347981 568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c437d9d466348f2863c407666c348aa" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:01.348115 1183 raft_consensus.cc:3037] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.348230 568 raft_consensus.cc:2454] T 3c437d9d466348f2863c407666c348aa P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:13:01.350386 30136 consensus_queue.cc:237] T b6c9fd3cbbd5453897aa072600db569d P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:01.350943 1183 raft_consensus.cc:504] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:01.352602 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c437d9d466348f2863c407666c348aa" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:13:01.352830 436 raft_consensus.cc:3037] T 3c437d9d466348f2863c407666c348aa P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.354692 1377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7527cc5931e24fa180efedbf81438efd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.355661 436 raft_consensus.cc:2454] T 3c437d9d466348f2863c407666c348aa P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:13:01.358511 29872 raft_consensus.cc:1261] T b6c9fd3cbbd5453897aa072600db569d P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:01.359977 32709 leader_election.cc:302] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:01.361001 1380 tablet_copy_client.cc:523] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:01.361845 30006 raft_consensus.cc:1261] T b6c9fd3cbbd5453897aa072600db569d P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:01.360616 1346 consensus_queue.cc:1035] T b6c9fd3cbbd5453897aa072600db569d P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20240503 02:13:01.364884 568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c437d9d466348f2863c407666c348aa" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:13:01.365110 568 raft_consensus.cc:3037] T 3c437d9d466348f2863c407666c348aa P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.366425 1380 tablet_bootstrap.cc:492] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
W20240503 02:13:01.367055 30070 consensus_peers.cc:485] T b6c9fd3cbbd5453897aa072600db569d P a9094240860e43bfb129193f3bfd5be6 -> Peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): Couldn't send request to peer fbbc2be6382e47908ecb7c68c367bd26. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6c9fd3cbbd5453897aa072600db569d. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:01.367571 912 consensus_queue.cc:1035] T b6c9fd3cbbd5453897aa072600db569d P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:01.367915 568 raft_consensus.cc:2454] T 3c437d9d466348f2863c407666c348aa P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:13:01.368811 1183 leader_election.cc:288] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Requested vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:13:01.369228 1183 raft_consensus.cc:2781] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:01.369678 1183 raft_consensus.cc:686] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Becoming Leader. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Running, Role: LEADER
I20240503 02:13:01.373322 1183 consensus_queue.cc:237] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:01.391523 29729 catalog_manager.cc:5484] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6). New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:01.398527 1377 tablet_copy_client.cc:783] T 7527cc5931e24fa180efedbf81438efd P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:01.402194 1377 tablet_copy_client.cc:647] T 7527cc5931e24fa180efedbf81438efd P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:01.405443 1377 tablet_copy_client.cc:523] T 7527cc5931e24fa180efedbf81438efd P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:01.407037 1346 raft_consensus.cc:2932] T b6c9fd3cbbd5453897aa072600db569d P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:01.409850 1376 tablet_bootstrap.cc:492] T 96de1c26ae884ea18b63b328db040ba1 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:01.410063 30004 raft_consensus.cc:2932] T b6c9fd3cbbd5453897aa072600db569d P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:01.414088 29872 raft_consensus.cc:2932] T b6c9fd3cbbd5453897aa072600db569d P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:01.414991 29729 catalog_manager.cc:5484] T b6c9fd3cbbd5453897aa072600db569d P a9094240860e43bfb129193f3bfd5be6 reported cstate change: config changed from index -1 to 4, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:01.415371 1376 tablet_bootstrap.cc:492] T 96de1c26ae884ea18b63b328db040ba1 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap complete.
I20240503 02:13:01.415697 1376 ts_tablet_manager.cc:1367] T 96de1c26ae884ea18b63b328db040ba1 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.114s user 0.041s sys 0.005s
I20240503 02:13:01.416543 1376 raft_consensus.cc:348] T 96de1c26ae884ea18b63b328db040ba1 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:01.418529 1376 raft_consensus.cc:729] T 96de1c26ae884ea18b63b328db040ba1 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.421738 1376 consensus_queue.cc:260] T 96de1c26ae884ea18b63b328db040ba1 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:01.433333 1376 ts_tablet_manager.cc:1398] T 96de1c26ae884ea18b63b328db040ba1 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.017s user 0.002s sys 0.001s
I20240503 02:13:01.437088 30286 tablet_copy_service.cc:342] P 1f10f80f13c84e80b03eda7085a3b012: Request end of tablet copy session fbbc2be6382e47908ecb7c68c367bd26-96de1c26ae884ea18b63b328db040ba1 received from {username='slave'} at 127.22.108.7:52333
I20240503 02:13:01.437271 30286 tablet_copy_service.cc:433] P 1f10f80f13c84e80b03eda7085a3b012: ending tablet copy session fbbc2be6382e47908ecb7c68c367bd26-96de1c26ae884ea18b63b328db040ba1 on tablet 96de1c26ae884ea18b63b328db040ba1 with peer fbbc2be6382e47908ecb7c68c367bd26
I20240503 02:13:01.438401 1348 tablet_bootstrap.cc:492] T 5f4b72b138c64ac58e6477cd2b1fb215 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:01.438895 1348 tablet_bootstrap.cc:492] T 5f4b72b138c64ac58e6477cd2b1fb215 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap complete.
I20240503 02:13:01.439194 1348 ts_tablet_manager.cc:1367] T 5f4b72b138c64ac58e6477cd2b1fb215 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.108s user 0.045s sys 0.016s
I20240503 02:13:01.440094 1348 raft_consensus.cc:348] T 5f4b72b138c64ac58e6477cd2b1fb215 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:01.440341 1348 raft_consensus.cc:729] T 5f4b72b138c64ac58e6477cd2b1fb215 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.442544 29731 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:13:01.444866 1377 tablet_bootstrap.cc:492] T 7527cc5931e24fa180efedbf81438efd P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:01.445878 1348 consensus_queue.cc:260] T 5f4b72b138c64ac58e6477cd2b1fb215 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:01.448607 1348 ts_tablet_manager.cc:1398] T 5f4b72b138c64ac58e6477cd2b1fb215 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.009s user 0.002s sys 0.001s
I20240503 02:13:01.450615 30004 raft_consensus.cc:1261] T 3b840a1964d049f8a9948a09c059a068 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:01.451217 307 raft_consensus.cc:3037] T 3b840a1964d049f8a9948a09c059a068 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.451244 1181 consensus_queue.cc:1035] T 3b840a1964d049f8a9948a09c059a068 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:01.454506 307 raft_consensus.cc:1261] T 3b840a1964d049f8a9948a09c059a068 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:01.458595 29895 tablet_copy_service.cc:342] P 16440e365430405fb10366ab70e2b429: Request end of tablet copy session 8a666e798fe8477d8dd1e51b385d5ead-5f4b72b138c64ac58e6477cd2b1fb215 received from {username='slave'} at 127.22.108.6:49099
I20240503 02:13:01.458845 29895 tablet_copy_service.cc:433] P 16440e365430405fb10366ab70e2b429: ending tablet copy session 8a666e798fe8477d8dd1e51b385d5ead-5f4b72b138c64ac58e6477cd2b1fb215 on tablet 5f4b72b138c64ac58e6477cd2b1fb215 with peer 8a666e798fe8477d8dd1e51b385d5ead
I20240503 02:13:01.459641 1181 consensus_queue.cc:1035] T 3b840a1964d049f8a9948a09c059a068 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:01.479130 1348 ts_tablet_manager.cc:902] T 08ee37b160d441ddabb640be907d6771 P 8a666e798fe8477d8dd1e51b385d5ead: Initiating tablet copy from peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:13:01.482419 778 raft_consensus.cc:482] T a52bee1fdc9d48b1a3463ecd62c8b44e P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:01.482641 778 raft_consensus.cc:504] T a52bee1fdc9d48b1a3463ecd62c8b44e P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:01.483482 778 leader_election.cc:288] T a52bee1fdc9d48b1a3463ecd62c8b44e P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:01.484302 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a52bee1fdc9d48b1a3463ecd62c8b44e" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:13:01.484570 436 raft_consensus.cc:2454] T a52bee1fdc9d48b1a3463ecd62c8b44e P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:13:01.485193 29731 catalog_manager.cc:5576] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975) does not have the latest schema for tablet 3b840a1964d049f8a9948a09c059a068 (table rebalancer_dynamic_table [id=c985f9c1d7a040d793cf0eaab4c1c6c5]). Expected version 1 got 0
I20240503 02:13:01.485450 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a52bee1fdc9d48b1a3463ecd62c8b44e" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:13:01.485688 305 raft_consensus.cc:2454] T a52bee1fdc9d48b1a3463ecd62c8b44e P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:13:01.486368 29940 leader_election.cc:302] T a52bee1fdc9d48b1a3463ecd62c8b44e P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:01.487147 778 raft_consensus.cc:2781] T a52bee1fdc9d48b1a3463ecd62c8b44e P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:01.487270 778 raft_consensus.cc:482] T a52bee1fdc9d48b1a3463ecd62c8b44e P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:01.487356 778 raft_consensus.cc:3037] T a52bee1fdc9d48b1a3463ecd62c8b44e P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.490993 778 raft_consensus.cc:504] T a52bee1fdc9d48b1a3463ecd62c8b44e P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:01.492751 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a52bee1fdc9d48b1a3463ecd62c8b44e" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:13:01.493047 436 raft_consensus.cc:3037] T a52bee1fdc9d48b1a3463ecd62c8b44e P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.496331 436 raft_consensus.cc:2454] T a52bee1fdc9d48b1a3463ecd62c8b44e P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:13:01.497732 29940 leader_election.cc:302] T a52bee1fdc9d48b1a3463ecd62c8b44e P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:01.498173 1319 raft_consensus.cc:2781] T a52bee1fdc9d48b1a3463ecd62c8b44e P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:01.506691 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a52bee1fdc9d48b1a3463ecd62c8b44e" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:13:01.506955 305 raft_consensus.cc:3037] T a52bee1fdc9d48b1a3463ecd62c8b44e P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.510618 305 raft_consensus.cc:2454] T a52bee1fdc9d48b1a3463ecd62c8b44e P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:13:01.511906 778 leader_election.cc:288] T a52bee1fdc9d48b1a3463ecd62c8b44e P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Requested vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:01.520694 1424 tablet.cc:1722] T 3b840a1964d049f8a9948a09c059a068 P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:01.521677 1424 tablet.cc:1620] T 3b840a1964d049f8a9948a09c059a068 P fbbc2be6382e47908ecb7c68c367bd26: MemRowSet was empty: no flush needed.
I20240503 02:13:01.525481 1425 tablet.cc:1722] T 3b840a1964d049f8a9948a09c059a068 P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:01.526068 1319 raft_consensus.cc:686] T a52bee1fdc9d48b1a3463ecd62c8b44e P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Becoming Leader. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Running, Role: LEADER
I20240503 02:13:01.526438 1425 tablet.cc:1620] T 3b840a1964d049f8a9948a09c059a068 P 8a666e798fe8477d8dd1e51b385d5ead: MemRowSet was empty: no flush needed.
I20240503 02:13:01.526433 1319 consensus_queue.cc:237] T a52bee1fdc9d48b1a3463ecd62c8b44e P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:01.530200 1427 ts_tablet_manager.cc:902] T 7076453cbe5944eb9b075127e1e0495d P 9a4d92bc25374fb3960fde7d10be309d: Initiating tablet copy from peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:13:01.534842 29731 catalog_manager.cc:5576] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975) does not have the latest schema for tablet 3b840a1964d049f8a9948a09c059a068 (table rebalancer_dynamic_table [id=c985f9c1d7a040d793cf0eaab4c1c6c5]). Expected version 1 got 0
I20240503 02:13:01.540529 1280 raft_consensus.cc:482] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:01.540728 1280 raft_consensus.cc:504] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:01.541500 1280 leader_election.cc:288] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:13:01.542387 568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bd4fcc4faf44d148a03beaafc1c05ba" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:01.544246 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bd4fcc4faf44d148a03beaafc1c05ba" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:13:01.544570 307 raft_consensus.cc:2379] T 6bd4fcc4faf44d148a03beaafc1c05ba P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 16440e365430405fb10366ab70e2b429 in current term 1: Already voted for candidate f575ce92c0224eeb80b9fae5f6561fbe in this term.
I20240503 02:13:01.545079 29809 leader_election.cc:302] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 16440e365430405fb10366ab70e2b429; no voters: 8a666e798fe8477d8dd1e51b385d5ead, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:13:01.545740 1280 raft_consensus.cc:3037] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.548691 1280 raft_consensus.cc:2726] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:01.550796 1427 tablet_copy_client.cc:308] T 7076453cbe5944eb9b075127e1e0495d P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.3:42679
I20240503 02:13:01.551563 30156 tablet_copy_service.cc:140] P a9094240860e43bfb129193f3bfd5be6: Received BeginTabletCopySession request for tablet 7076453cbe5944eb9b075127e1e0495d from peer 9a4d92bc25374fb3960fde7d10be309d ({username='slave'} at 127.22.108.5:33825)
I20240503 02:13:01.551774 30156 tablet_copy_service.cc:161] P a9094240860e43bfb129193f3bfd5be6: Beginning new tablet copy session on tablet 7076453cbe5944eb9b075127e1e0495d from peer 9a4d92bc25374fb3960fde7d10be309d at {username='slave'} at 127.22.108.5:33825: session id = 9a4d92bc25374fb3960fde7d10be309d-7076453cbe5944eb9b075127e1e0495d
I20240503 02:13:01.553620 29731 catalog_manager.cc:5576] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873) does not have the latest schema for tablet 3b840a1964d049f8a9948a09c059a068 (table rebalancer_dynamic_table [id=c985f9c1d7a040d793cf0eaab4c1c6c5]). Expected version 1 got 0
I20240503 02:13:01.554872 30156 tablet_copy_source_session.cc:205] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:01.556670 1427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7076453cbe5944eb9b075127e1e0495d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.557482 1348 tablet_copy_client.cc:308] T 08ee37b160d441ddabb640be907d6771 P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.2:37873
I20240503 02:13:01.557931 29729 catalog_manager.cc:5484] T a52bee1fdc9d48b1a3463ecd62c8b44e P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: term changed from 0 to 1, leader changed from <none> to 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2). New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:01.558223 30026 tablet_copy_service.cc:140] P 155735c7f8b14997ac4c770aee2a9401: Received BeginTabletCopySession request for tablet 08ee37b160d441ddabb640be907d6771 from peer 8a666e798fe8477d8dd1e51b385d5ead ({username='slave'} at 127.22.108.6:33785)
I20240503 02:13:01.558393 30026 tablet_copy_service.cc:161] P 155735c7f8b14997ac4c770aee2a9401: Beginning new tablet copy session on tablet 08ee37b160d441ddabb640be907d6771 from peer 8a666e798fe8477d8dd1e51b385d5ead at {username='slave'} at 127.22.108.6:33785: session id = 8a666e798fe8477d8dd1e51b385d5ead-08ee37b160d441ddabb640be907d6771
I20240503 02:13:01.568630 30026 tablet_copy_source_session.cc:205] T 08ee37b160d441ddabb640be907d6771 P 155735c7f8b14997ac4c770aee2a9401: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:01.572417 1348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08ee37b160d441ddabb640be907d6771. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.575618 1439 tablet.cc:1722] T 3b840a1964d049f8a9948a09c059a068 P 155735c7f8b14997ac4c770aee2a9401: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:01.576308 1439 tablet.cc:1620] T 3b840a1964d049f8a9948a09c059a068 P 155735c7f8b14997ac4c770aee2a9401: MemRowSet was empty: no flush needed.
I20240503 02:13:01.577687 1427 tablet_copy_client.cc:783] T 7076453cbe5944eb9b075127e1e0495d P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:01.578011 1427 tablet_copy_client.cc:647] T 7076453cbe5944eb9b075127e1e0495d P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:01.580041 29731 catalog_manager.cc:5576] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873) does not have the latest schema for tablet 3b840a1964d049f8a9948a09c059a068 (table rebalancer_dynamic_table [id=c985f9c1d7a040d793cf0eaab4c1c6c5]). Expected version 1 got 0
I20240503 02:13:01.580550 1427 tablet_copy_client.cc:523] T 7076453cbe5944eb9b075127e1e0495d P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:01.582705 1372 consensus_queue.cc:1035] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:01.586035 1348 tablet_copy_client.cc:783] T 08ee37b160d441ddabb640be907d6771 P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:01.586277 1348 tablet_copy_client.cc:647] T 08ee37b160d441ddabb640be907d6771 P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:01.594321 29729 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=c985f9c1d7a040d793cf0eaab4c1c6c5] alter complete (version 1)
I20240503 02:13:01.594980 1427 tablet_bootstrap.cc:492] T 7076453cbe5944eb9b075127e1e0495d P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:01.597295 1348 tablet_copy_client.cc:523] T 08ee37b160d441ddabb640be907d6771 P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:01.601081 1348 tablet_bootstrap.cc:492] T 08ee37b160d441ddabb640be907d6771 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:01.604506 1380 tablet_bootstrap.cc:492] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:01.608798 1380 tablet_bootstrap.cc:492] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap complete.
I20240503 02:13:01.609218 1380 ts_tablet_manager.cc:1367] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.243s user 0.061s sys 0.008s
I20240503 02:13:01.609935 30264 raft_consensus.cc:1203] T 83c7d1060ccf49a2adda4302600a9876 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:01.615000 1380 raft_consensus.cc:348] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:01.615466 1380 raft_consensus.cc:729] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.616273 1380 consensus_queue.cc:260] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:01.618877 1380 ts_tablet_manager.cc:1398] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 02:13:01.619516 1113 consensus_queue.cc:1035] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:01.620033 30156 tablet_copy_service.cc:342] P a9094240860e43bfb129193f3bfd5be6: Request end of tablet copy session 9a4d92bc25374fb3960fde7d10be309d-8fdbaf1df10b46d5a47326415d0cd0e9 received from {username='slave'} at 127.22.108.5:33825
I20240503 02:13:01.620195 30156 tablet_copy_service.cc:433] P a9094240860e43bfb129193f3bfd5be6: ending tablet copy session 9a4d92bc25374fb3960fde7d10be309d-8fdbaf1df10b46d5a47326415d0cd0e9 on tablet 8fdbaf1df10b46d5a47326415d0cd0e9 with peer 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:13:01.654832 1377 tablet_bootstrap.cc:492] T 7527cc5931e24fa180efedbf81438efd P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:01.655457 1377 tablet_bootstrap.cc:492] T 7527cc5931e24fa180efedbf81438efd P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap complete.
I20240503 02:13:01.655823 1377 ts_tablet_manager.cc:1367] T 7527cc5931e24fa180efedbf81438efd P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.211s user 0.053s sys 0.008s
I20240503 02:13:01.656793 1377 raft_consensus.cc:348] T 7527cc5931e24fa180efedbf81438efd P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:01.657188 1377 raft_consensus.cc:729] T 7527cc5931e24fa180efedbf81438efd P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.657531 1377 consensus_queue.cc:260] T 7527cc5931e24fa180efedbf81438efd P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:01.658452 1377 ts_tablet_manager.cc:1398] T 7527cc5931e24fa180efedbf81438efd P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:01.659786 30286 tablet_copy_service.cc:342] P 1f10f80f13c84e80b03eda7085a3b012: Request end of tablet copy session f575ce92c0224eeb80b9fae5f6561fbe-7527cc5931e24fa180efedbf81438efd received from {username='slave'} at 127.22.108.8:51621
I20240503 02:13:01.659927 30286 tablet_copy_service.cc:433] P 1f10f80f13c84e80b03eda7085a3b012: ending tablet copy session f575ce92c0224eeb80b9fae5f6561fbe-7527cc5931e24fa180efedbf81438efd on tablet 7527cc5931e24fa180efedbf81438efd with peer f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:13:01.661893 307 raft_consensus.cc:1203] T 5f4b72b138c64ac58e6477cd2b1fb215 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:01.686486 1348 tablet_bootstrap.cc:492] T 08ee37b160d441ddabb640be907d6771 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:01.686944 1348 tablet_bootstrap.cc:492] T 08ee37b160d441ddabb640be907d6771 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap complete.
I20240503 02:13:01.687191 1348 ts_tablet_manager.cc:1367] T 08ee37b160d441ddabb640be907d6771 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.086s user 0.041s sys 0.025s
I20240503 02:13:01.687351 1427 tablet_bootstrap.cc:492] T 7076453cbe5944eb9b075127e1e0495d P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:01.687359 568 raft_consensus.cc:1203] T 7527cc5931e24fa180efedbf81438efd P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:01.687794 1427 tablet_bootstrap.cc:492] T 7076453cbe5944eb9b075127e1e0495d P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap complete.
I20240503 02:13:01.687951 1348 raft_consensus.cc:348] T 08ee37b160d441ddabb640be907d6771 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:01.688115 1427 ts_tablet_manager.cc:1367] T 7076453cbe5944eb9b075127e1e0495d P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.093s user 0.046s sys 0.004s
I20240503 02:13:01.688207 1348 raft_consensus.cc:729] T 08ee37b160d441ddabb640be907d6771 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.688477 1348 consensus_queue.cc:260] T 08ee37b160d441ddabb640be907d6771 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:01.689018 1348 ts_tablet_manager.cc:1398] T 08ee37b160d441ddabb640be907d6771 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:13:01.688900 1427 raft_consensus.cc:348] T 7076453cbe5944eb9b075127e1e0495d P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:01.689167 1427 raft_consensus.cc:729] T 7076453cbe5944eb9b075127e1e0495d P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.689435 1427 consensus_queue.cc:260] T 7076453cbe5944eb9b075127e1e0495d P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:01.689957 1427 ts_tablet_manager.cc:1398] T 7076453cbe5944eb9b075127e1e0495d P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 02:13:01.690711 30156 tablet_copy_service.cc:342] P a9094240860e43bfb129193f3bfd5be6: Request end of tablet copy session 9a4d92bc25374fb3960fde7d10be309d-7076453cbe5944eb9b075127e1e0495d received from {username='slave'} at 127.22.108.5:33825
I20240503 02:13:01.690860 30156 tablet_copy_service.cc:433] P a9094240860e43bfb129193f3bfd5be6: ending tablet copy session 9a4d92bc25374fb3960fde7d10be309d-7076453cbe5944eb9b075127e1e0495d on tablet 7076453cbe5944eb9b075127e1e0495d with peer 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:13:01.691219 30026 tablet_copy_service.cc:342] P 155735c7f8b14997ac4c770aee2a9401: Request end of tablet copy session 8a666e798fe8477d8dd1e51b385d5ead-08ee37b160d441ddabb640be907d6771 received from {username='slave'} at 127.22.108.6:33785
I20240503 02:13:01.691386 30026 tablet_copy_service.cc:433] P 155735c7f8b14997ac4c770aee2a9401: ending tablet copy session 8a666e798fe8477d8dd1e51b385d5ead-08ee37b160d441ddabb640be907d6771 on tablet 08ee37b160d441ddabb640be907d6771 with peer 8a666e798fe8477d8dd1e51b385d5ead
I20240503 02:13:01.694444 436 raft_consensus.cc:1203] T 96de1c26ae884ea18b63b328db040ba1 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:01.717300 29729 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:13:01.731998 1424 tablet.cc:1722] T 3b840a1964d049f8a9948a09c059a068 P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:01.732520 1424 tablet.cc:1620] T 3b840a1964d049f8a9948a09c059a068 P fbbc2be6382e47908ecb7c68c367bd26: MemRowSet was empty: no flush needed.
I20240503 02:13:01.736729 1425 tablet.cc:1722] T 3b840a1964d049f8a9948a09c059a068 P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:01.737884 1425 tablet.cc:1620] T 3b840a1964d049f8a9948a09c059a068 P 8a666e798fe8477d8dd1e51b385d5ead: MemRowSet was empty: no flush needed.
I20240503 02:13:01.739223 1439 tablet.cc:1722] T 3b840a1964d049f8a9948a09c059a068 P 155735c7f8b14997ac4c770aee2a9401: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:01.739718 1439 tablet.cc:1620] T 3b840a1964d049f8a9948a09c059a068 P 155735c7f8b14997ac4c770aee2a9401: MemRowSet was empty: no flush needed.
I20240503 02:13:01.745482 29725 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=c985f9c1d7a040d793cf0eaab4c1c6c5] alter complete (version 2)
I20240503 02:13:01.751206 32640 raft_consensus.cc:1203] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:01.806202 1410 consensus_queue.cc:1035] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:01.811302 1183 consensus_queue.cc:1035] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:01.855965 29731 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:13:01.866258 1424 tablet.cc:1722] T 3b840a1964d049f8a9948a09c059a068 P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:01.869765 1439 tablet.cc:1722] T 3b840a1964d049f8a9948a09c059a068 P 155735c7f8b14997ac4c770aee2a9401: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:01.873901 1425 tablet.cc:1722] T 3b840a1964d049f8a9948a09c059a068 P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:01.898581 29731 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table_new_name [id=c985f9c1d7a040d793cf0eaab4c1c6c5] alter complete (version 3)
I20240503 02:13:01.918129 29986 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:01.918916 1376 ts_tablet_manager.cc:902] T b6c9fd3cbbd5453897aa072600db569d P fbbc2be6382e47908ecb7c68c367bd26: Initiating tablet copy from peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:13:01.954479 29853 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:13:01.966960 1319 consensus_queue.cc:1035] T a52bee1fdc9d48b1a3463ecd62c8b44e P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:01.987694 30245 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:02.004525 29730 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:02.004817 29730 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:02.006645 778 consensus_queue.cc:1035] T a52bee1fdc9d48b1a3463ecd62c8b44e P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:02.007573 32640 raft_consensus.cc:1203] T 7076453cbe5944eb9b075127e1e0495d P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:02.013226 29730 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet 3b840a1964d049f8a9948a09c059a068
I20240503 02:13:02.014501 32755 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b840a1964d049f8a9948a09c059a068 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:02 UTC) from {username='slave'} at 127.0.0.1:46360
I20240503 02:13:02.014782 29986 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b840a1964d049f8a9948a09c059a068 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:02 UTC) from {username='slave'} at 127.0.0.1:52158
I20240503 02:13:02.024086 32620 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:02.024945 547 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:13:02.025462 1376 tablet_copy_client.cc:308] T b6c9fd3cbbd5453897aa072600db569d P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.3:42679
I20240503 02:13:02.036940 307 raft_consensus.cc:1203] T 08ee37b160d441ddabb640be907d6771 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:02.046123 1509 tablet_replica.cc:330] T 3b840a1964d049f8a9948a09c059a068 P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:13:02.049502 1509 raft_consensus.cc:2229] T 3b840a1964d049f8a9948a09c059a068 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:02.049943 32752 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:13:02.050796 1509 raft_consensus.cc:2258] T 3b840a1964d049f8a9948a09c059a068 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:02.053277 1509 ts_tablet_manager.cc:1875] T 3b840a1964d049f8a9948a09c059a068 P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:02.060411 1509 ts_tablet_manager.cc:1888] T 3b840a1964d049f8a9948a09c059a068 P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:02.069793 1509 log.cc:1196] T 3b840a1964d049f8a9948a09c059a068 P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/3b840a1964d049f8a9948a09c059a068
I20240503 02:13:02.070492 1509 ts_tablet_manager.cc:1909] T 3b840a1964d049f8a9948a09c059a068 P 155735c7f8b14997ac4c770aee2a9401: Deleting consensus metadata
I20240503 02:13:02.070853 30114 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:02.072255 1508 tablet_replica.cc:330] T 3b840a1964d049f8a9948a09c059a068 P 8a666e798fe8477d8dd1e51b385d5ead: stopping tablet replica
I20240503 02:13:02.072360 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet 3b840a1964d049f8a9948a09c059a068 (table rebalancer_dynamic_table_new_name [id=c985f9c1d7a040d793cf0eaab4c1c6c5]) successfully deleted
I20240503 02:13:02.072736 1508 raft_consensus.cc:2229] T 3b840a1964d049f8a9948a09c059a068 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:02.077271 1508 raft_consensus.cc:2258] T 3b840a1964d049f8a9948a09c059a068 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:02.081073 29730 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_extra_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:02.084185 30156 tablet_copy_service.cc:140] P a9094240860e43bfb129193f3bfd5be6: Received BeginTabletCopySession request for tablet b6c9fd3cbbd5453897aa072600db569d from peer fbbc2be6382e47908ecb7c68c367bd26 ({username='slave'} at 127.22.108.7:37671)
I20240503 02:13:02.084379 30156 tablet_copy_service.cc:161] P a9094240860e43bfb129193f3bfd5be6: Beginning new tablet copy session on tablet b6c9fd3cbbd5453897aa072600db569d from peer fbbc2be6382e47908ecb7c68c367bd26 at {username='slave'} at 127.22.108.7:37671: session id = fbbc2be6382e47908ecb7c68c367bd26-b6c9fd3cbbd5453897aa072600db569d
I20240503 02:13:02.087473 30156 tablet_copy_source_session.cc:205] T b6c9fd3cbbd5453897aa072600db569d P a9094240860e43bfb129193f3bfd5be6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:02.088816 1376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6c9fd3cbbd5453897aa072600db569d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:02.097285 1508 ts_tablet_manager.cc:1875] T 3b840a1964d049f8a9948a09c059a068 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:02.098803 30245 tablet_service.cc:1467] Processing CreateTablet for tablet cf0743dee35b411e8fd069322d4f3da6 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=ff2eb02eb88b44a0b0da2abc6307220a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:02.099018 32620 tablet_service.cc:1467] Processing CreateTablet for tablet cf0743dee35b411e8fd069322d4f3da6 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=ff2eb02eb88b44a0b0da2abc6307220a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:02.099318 30245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf0743dee35b411e8fd069322d4f3da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:02.099416 32620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf0743dee35b411e8fd069322d4f3da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:02.103915 32621 tablet_service.cc:1467] Processing CreateTablet for tablet 2baf70e0e1894cea9fdeb92279bc0cff (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=ff2eb02eb88b44a0b0da2abc6307220a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:02.104115 32619 tablet_service.cc:1467] Processing CreateTablet for tablet 331905034afd48c19eb2bc2e2ec40280 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=ff2eb02eb88b44a0b0da2abc6307220a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:02.104229 29986 tablet_service.cc:1467] Processing CreateTablet for tablet 331905034afd48c19eb2bc2e2ec40280 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=ff2eb02eb88b44a0b0da2abc6307220a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:02.104382 32621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2baf70e0e1894cea9fdeb92279bc0cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:02.104625 29986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 331905034afd48c19eb2bc2e2ec40280. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:02.105170 32752 tablet_service.cc:1467] Processing CreateTablet for tablet 331905034afd48c19eb2bc2e2ec40280 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=ff2eb02eb88b44a0b0da2abc6307220a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:02.105616 32752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 331905034afd48c19eb2bc2e2ec40280. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:02.111294 32619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 331905034afd48c19eb2bc2e2ec40280. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:02.116439 1376 tablet_copy_client.cc:783] T b6c9fd3cbbd5453897aa072600db569d P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:02.116686 1376 tablet_copy_client.cc:647] T b6c9fd3cbbd5453897aa072600db569d P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:02.118214 29730 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:13:02.127456 1376 tablet_copy_client.cc:523] T b6c9fd3cbbd5453897aa072600db569d P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:02.131283 32754 tablet_service.cc:1467] Processing CreateTablet for tablet d7ef72cb49c64085a6d5b04228e275e4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=42b5416694924d77976fd377a8418575]), partition=
I20240503 02:13:02.131821 32754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7ef72cb49c64085a6d5b04228e275e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:02.131799 30244 tablet_service.cc:1467] Processing CreateTablet for tablet d7ef72cb49c64085a6d5b04228e275e4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=42b5416694924d77976fd377a8418575]), partition=
I20240503 02:13:02.132150 30244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7ef72cb49c64085a6d5b04228e275e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:02.143182 32755 tablet_service.cc:1467] Processing CreateTablet for tablet 2baf70e0e1894cea9fdeb92279bc0cff (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=ff2eb02eb88b44a0b0da2abc6307220a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:02.143330 1514 tablet_bootstrap.cc:492] T 331905034afd48c19eb2bc2e2ec40280 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:13:02.143983 32755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2baf70e0e1894cea9fdeb92279bc0cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:02.145227 1514 tablet_bootstrap.cc:654] T 331905034afd48c19eb2bc2e2ec40280 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:02.149693 1513 tablet_bootstrap.cc:492] T cf0743dee35b411e8fd069322d4f3da6 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:13:02.153930 415 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b840a1964d049f8a9948a09c059a068 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:02 UTC) from {username='slave'} at 127.0.0.1:50814
I20240503 02:13:02.154619 1513 tablet_bootstrap.cc:654] T cf0743dee35b411e8fd069322d4f3da6 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:02.154994 1376 tablet_bootstrap.cc:492] T b6c9fd3cbbd5453897aa072600db569d P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:02.168004 1513 tablet_bootstrap.cc:492] T cf0743dee35b411e8fd069322d4f3da6 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:13:02.168186 1513 ts_tablet_manager.cc:1367] T cf0743dee35b411e8fd069322d4f3da6 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.006s
I20240503 02:13:02.170101 1513 raft_consensus.cc:348] T cf0743dee35b411e8fd069322d4f3da6 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:02.170089 416 tablet_service.cc:1467] Processing CreateTablet for tablet cf0743dee35b411e8fd069322d4f3da6 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=ff2eb02eb88b44a0b0da2abc6307220a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:02.170315 1513 raft_consensus.cc:374] T cf0743dee35b411e8fd069322d4f3da6 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:02.170403 1513 raft_consensus.cc:729] T cf0743dee35b411e8fd069322d4f3da6 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:13:02.170763 416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf0743dee35b411e8fd069322d4f3da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:02.171581 1513 consensus_queue.cc:260] T cf0743dee35b411e8fd069322d4f3da6 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:02.172160 1513 ts_tablet_manager.cc:1398] T cf0743dee35b411e8fd069322d4f3da6 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:13:02.174623 1513 tablet_bootstrap.cc:492] T d7ef72cb49c64085a6d5b04228e275e4 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:13:02.180001 1513 tablet_bootstrap.cc:654] T d7ef72cb49c64085a6d5b04228e275e4 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:02.186875 417 tablet_service.cc:1467] Processing CreateTablet for tablet 2baf70e0e1894cea9fdeb92279bc0cff (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=ff2eb02eb88b44a0b0da2abc6307220a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:02.187434 417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2baf70e0e1894cea9fdeb92279bc0cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:02.187970 1513 tablet_bootstrap.cc:492] T d7ef72cb49c64085a6d5b04228e275e4 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:13:02.188175 1513 ts_tablet_manager.cc:1367] T d7ef72cb49c64085a6d5b04228e275e4 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.005s
I20240503 02:13:02.190958 1513 raft_consensus.cc:348] T d7ef72cb49c64085a6d5b04228e275e4 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:02.191751 1513 raft_consensus.cc:374] T d7ef72cb49c64085a6d5b04228e275e4 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:02.192155 1513 raft_consensus.cc:729] T d7ef72cb49c64085a6d5b04228e275e4 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:13:02.192759 1513 consensus_queue.cc:260] T d7ef72cb49c64085a6d5b04228e275e4 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:02.195600 1513 ts_tablet_manager.cc:1398] T d7ef72cb49c64085a6d5b04228e275e4 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240503 02:13:02.208204 417 tablet_service.cc:1467] Processing CreateTablet for tablet d7ef72cb49c64085a6d5b04228e275e4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=42b5416694924d77976fd377a8418575]), partition=
I20240503 02:13:02.211958 417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7ef72cb49c64085a6d5b04228e275e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:02.218358 1521 tablet_bootstrap.cc:492] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:02.220834 1515 tablet_bootstrap.cc:492] T 331905034afd48c19eb2bc2e2ec40280 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:02.222299 1517 tablet_replica.cc:330] T 3b840a1964d049f8a9948a09c059a068 P fbbc2be6382e47908ecb7c68c367bd26: stopping tablet replica
I20240503 02:13:02.222718 1517 raft_consensus.cc:2229] T 3b840a1964d049f8a9948a09c059a068 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:02.232097 1517 raft_consensus.cc:2258] T 3b840a1964d049f8a9948a09c059a068 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:02.242271 1453 raft_consensus.cc:482] T cf0743dee35b411e8fd069322d4f3da6 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:02.242465 1453 raft_consensus.cc:504] T cf0743dee35b411e8fd069322d4f3da6 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:02.243242 1453 leader_election.cc:288] T cf0743dee35b411e8fd069322d4f3da6 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:02.243832 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf0743dee35b411e8fd069322d4f3da6" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:13:02.245692 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf0743dee35b411e8fd069322d4f3da6" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
W20240503 02:13:02.247116 30200 leader_election.cc:341] T cf0743dee35b411e8fd069322d4f3da6 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:13:02.247928 30200 leader_election.cc:341] T cf0743dee35b411e8fd069322d4f3da6 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:02.248092 30200 leader_election.cc:302] T cf0743dee35b411e8fd069322d4f3da6 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012; no voters: 9a4d92bc25374fb3960fde7d10be309d, fbbc2be6382e47908ecb7c68c367bd26
I20240503 02:13:02.248615 1453 raft_consensus.cc:2726] T cf0743dee35b411e8fd069322d4f3da6 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:02.253110 1517 ts_tablet_manager.cc:1875] T 3b840a1964d049f8a9948a09c059a068 P fbbc2be6382e47908ecb7c68c367bd26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:02.270772 1521 tablet_bootstrap.cc:654] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:02.273751 1514 tablet_bootstrap.cc:492] T 331905034afd48c19eb2bc2e2ec40280 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:13:02.274142 1514 ts_tablet_manager.cc:1367] T 331905034afd48c19eb2bc2e2ec40280 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.131s user 0.026s sys 0.062s
I20240503 02:13:02.275085 1514 raft_consensus.cc:348] T 331905034afd48c19eb2bc2e2ec40280 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:02.277328 1514 raft_consensus.cc:374] T 331905034afd48c19eb2bc2e2ec40280 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:02.277487 1514 raft_consensus.cc:729] T 331905034afd48c19eb2bc2e2ec40280 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:13:02.277818 1514 consensus_queue.cc:260] T 331905034afd48c19eb2bc2e2ec40280 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:02.278322 1512 tablet_bootstrap.cc:492] T cf0743dee35b411e8fd069322d4f3da6 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:02.278825 1514 ts_tablet_manager.cc:1398] T 331905034afd48c19eb2bc2e2ec40280 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:13:02.279187 415 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:02.280751 1512 tablet_bootstrap.cc:654] T cf0743dee35b411e8fd069322d4f3da6 P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:02.290302 1515 tablet_bootstrap.cc:654] T 331905034afd48c19eb2bc2e2ec40280 P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:02.296479 1319 raft_consensus.cc:482] T 331905034afd48c19eb2bc2e2ec40280 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:02.296676 1319 raft_consensus.cc:504] T 331905034afd48c19eb2bc2e2ec40280 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:02.297657 1319 leader_election.cc:288] T 331905034afd48c19eb2bc2e2ec40280 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:02.298312 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331905034afd48c19eb2bc2e2ec40280" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:13:02.298823 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331905034afd48c19eb2bc2e2ec40280" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
W20240503 02:13:02.299626 29940 leader_election.cc:341] T 331905034afd48c19eb2bc2e2ec40280 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:13:02.299840 29940 leader_election.cc:341] T 331905034afd48c19eb2bc2e2ec40280 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:02.299926 29940 leader_election.cc:302] T 331905034afd48c19eb2bc2e2ec40280 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401; no voters: 8a666e798fe8477d8dd1e51b385d5ead, 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:13:02.300453 1428 raft_consensus.cc:2726] T 331905034afd48c19eb2bc2e2ec40280 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:02.305128 1515 tablet_bootstrap.cc:492] T 331905034afd48c19eb2bc2e2ec40280 P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:02.305367 1515 ts_tablet_manager.cc:1367] T 331905034afd48c19eb2bc2e2ec40280 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.085s user 0.008s sys 0.004s
I20240503 02:13:02.306353 1515 raft_consensus.cc:348] T 331905034afd48c19eb2bc2e2ec40280 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:02.306645 1515 raft_consensus.cc:374] T 331905034afd48c19eb2bc2e2ec40280 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:02.306771 1515 raft_consensus.cc:729] T 331905034afd48c19eb2bc2e2ec40280 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:02.306882 1521 tablet_bootstrap.cc:492] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:02.307090 1521 ts_tablet_manager.cc:1367] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.089s user 0.005s sys 0.000s
I20240503 02:13:02.307106 1515 consensus_queue.cc:260] T 331905034afd48c19eb2bc2e2ec40280 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:02.308645 1521 raft_consensus.cc:348] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:02.309017 1521 raft_consensus.cc:374] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:02.309226 1521 raft_consensus.cc:729] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:02.309661 1521 consensus_queue.cc:260] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:02.310688 1515 ts_tablet_manager.cc:1398] T 331905034afd48c19eb2bc2e2ec40280 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:13:02.311025 1515 tablet_bootstrap.cc:492] T d7ef72cb49c64085a6d5b04228e275e4 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:02.314147 1515 tablet_bootstrap.cc:654] T d7ef72cb49c64085a6d5b04228e275e4 P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:02.315925 1515 tablet_bootstrap.cc:492] T d7ef72cb49c64085a6d5b04228e275e4 P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:02.316083 1515 ts_tablet_manager.cc:1367] T d7ef72cb49c64085a6d5b04228e275e4 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240503 02:13:02.318235 1515 raft_consensus.cc:348] T d7ef72cb49c64085a6d5b04228e275e4 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:02.318570 1515 raft_consensus.cc:374] T d7ef72cb49c64085a6d5b04228e275e4 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:02.318755 1515 raft_consensus.cc:729] T d7ef72cb49c64085a6d5b04228e275e4 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:02.319124 1515 consensus_queue.cc:260] T d7ef72cb49c64085a6d5b04228e275e4 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:02.322840 1515 ts_tablet_manager.cc:1398] T d7ef72cb49c64085a6d5b04228e275e4 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20240503 02:13:02.325883 1515 tablet_bootstrap.cc:492] T 2baf70e0e1894cea9fdeb92279bc0cff P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:02.327519 1515 tablet_bootstrap.cc:654] T 2baf70e0e1894cea9fdeb92279bc0cff P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:02.331799 1508 ts_tablet_manager.cc:1888] T 3b840a1964d049f8a9948a09c059a068 P 8a666e798fe8477d8dd1e51b385d5ead: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:02.331938 1508 log.cc:1196] T 3b840a1964d049f8a9948a09c059a068 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal/wals/3b840a1964d049f8a9948a09c059a068
I20240503 02:13:02.332443 1508 ts_tablet_manager.cc:1909] T 3b840a1964d049f8a9948a09c059a068 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting consensus metadata
I20240503 02:13:02.333492 29719 catalog_manager.cc:4830] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): tablet 3b840a1964d049f8a9948a09c059a068 (table rebalancer_dynamic_table_new_name [id=c985f9c1d7a040d793cf0eaab4c1c6c5]) successfully deleted
I20240503 02:13:02.337077 1512 tablet_bootstrap.cc:492] T cf0743dee35b411e8fd069322d4f3da6 P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:02.337615 1521 ts_tablet_manager.cc:1398] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.030s user 0.003s sys 0.000s
I20240503 02:13:02.337952 1521 tablet_bootstrap.cc:492] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:02.341094 1512 ts_tablet_manager.cc:1367] T cf0743dee35b411e8fd069322d4f3da6 P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.063s user 0.003s sys 0.001s
I20240503 02:13:02.341955 1512 raft_consensus.cc:348] T cf0743dee35b411e8fd069322d4f3da6 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:02.342308 1512 raft_consensus.cc:374] T cf0743dee35b411e8fd069322d4f3da6 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:02.342428 1512 raft_consensus.cc:729] T cf0743dee35b411e8fd069322d4f3da6 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:02.342770 1517 ts_tablet_manager.cc:1888] T 3b840a1964d049f8a9948a09c059a068 P fbbc2be6382e47908ecb7c68c367bd26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:02.342701 1512 consensus_queue.cc:260] T cf0743dee35b411e8fd069322d4f3da6 P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:02.342900 1517 log.cc:1196] T 3b840a1964d049f8a9948a09c059a068 P fbbc2be6382e47908ecb7c68c367bd26: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal/wals/3b840a1964d049f8a9948a09c059a068
I20240503 02:13:02.344292 1512 ts_tablet_manager.cc:1398] T cf0743dee35b411e8fd069322d4f3da6 P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 02:13:02.344664 1512 tablet_bootstrap.cc:492] T 2baf70e0e1894cea9fdeb92279bc0cff P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:02.346736 1521 tablet_bootstrap.cc:654] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:02.348273 1512 tablet_bootstrap.cc:654] T 2baf70e0e1894cea9fdeb92279bc0cff P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:02.349345 1517 ts_tablet_manager.cc:1909] T 3b840a1964d049f8a9948a09c059a068 P fbbc2be6382e47908ecb7c68c367bd26: Deleting consensus metadata
I20240503 02:13:02.351078 29719 catalog_manager.cc:4830] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): tablet 3b840a1964d049f8a9948a09c059a068 (table rebalancer_dynamic_table_new_name [id=c985f9c1d7a040d793cf0eaab4c1c6c5]) successfully deleted
I20240503 02:13:02.352255 1515 tablet_bootstrap.cc:492] T 2baf70e0e1894cea9fdeb92279bc0cff P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:02.352463 1515 ts_tablet_manager.cc:1367] T 2baf70e0e1894cea9fdeb92279bc0cff P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.027s user 0.011s sys 0.001s
I20240503 02:13:02.353607 1515 raft_consensus.cc:348] T 2baf70e0e1894cea9fdeb92279bc0cff P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:02.353893 1515 raft_consensus.cc:374] T 2baf70e0e1894cea9fdeb92279bc0cff P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:02.354058 1515 raft_consensus.cc:729] T 2baf70e0e1894cea9fdeb92279bc0cff P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:02.354355 1515 consensus_queue.cc:260] T 2baf70e0e1894cea9fdeb92279bc0cff P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:02.358254 1515 ts_tablet_manager.cc:1398] T 2baf70e0e1894cea9fdeb92279bc0cff P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:13:02.353930 1352 raft_consensus.cc:482] T cf0743dee35b411e8fd069322d4f3da6 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:02.359719 1352 raft_consensus.cc:504] T cf0743dee35b411e8fd069322d4f3da6 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:02.360679 1352 leader_election.cc:288] T cf0743dee35b411e8fd069322d4f3da6 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:02.370347 1453 raft_consensus.cc:482] T d7ef72cb49c64085a6d5b04228e275e4 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:02.370509 1453 raft_consensus.cc:504] T d7ef72cb49c64085a6d5b04228e275e4 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:02.371156 1453 leader_election.cc:288] T d7ef72cb49c64085a6d5b04228e275e4 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:02.371691 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7ef72cb49c64085a6d5b04228e275e4" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
W20240503 02:13:02.372437 30200 leader_election.cc:341] T d7ef72cb49c64085a6d5b04228e275e4 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:02.380077 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf0743dee35b411e8fd069322d4f3da6" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:13:02.380468 436 raft_consensus.cc:2454] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 0.
I20240503 02:13:02.381296 32574 leader_election.cc:302] T cf0743dee35b411e8fd069322d4f3da6 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:02.381841 1352 raft_consensus.cc:2781] T cf0743dee35b411e8fd069322d4f3da6 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:02.381968 1352 raft_consensus.cc:482] T cf0743dee35b411e8fd069322d4f3da6 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:02.382068 1352 raft_consensus.cc:3037] T cf0743dee35b411e8fd069322d4f3da6 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:02.386268 1352 raft_consensus.cc:504] T cf0743dee35b411e8fd069322d4f3da6 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:02.387249 1352 leader_election.cc:288] T cf0743dee35b411e8fd069322d4f3da6 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 election: Requested vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:02.387984 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf0743dee35b411e8fd069322d4f3da6" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:13:02.388218 436 raft_consensus.cc:3037] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:02.388394 1521 tablet_bootstrap.cc:492] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:02.388590 1521 ts_tablet_manager.cc:1367] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.051s user 0.002s sys 0.005s
I20240503 02:13:02.389869 1521 raft_consensus.cc:348] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:02.390182 1521 raft_consensus.cc:374] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:02.390278 1521 raft_consensus.cc:729] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:02.390667 1521 consensus_queue.cc:260] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:02.391204 436 raft_consensus.cc:2454] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 1.
I20240503 02:13:02.392553 1521 ts_tablet_manager.cc:1398] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:13:02.392859 1521 tablet_bootstrap.cc:492] T d7ef72cb49c64085a6d5b04228e275e4 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:02.394551 1521 tablet_bootstrap.cc:654] T d7ef72cb49c64085a6d5b04228e275e4 P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:02.406069 32574 leader_election.cc:302] T cf0743dee35b411e8fd069322d4f3da6 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:02.406047 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7ef72cb49c64085a6d5b04228e275e4" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:13:02.406373 305 raft_consensus.cc:2454] T d7ef72cb49c64085a6d5b04228e275e4 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:13:02.406503 1352 raft_consensus.cc:2781] T cf0743dee35b411e8fd069322d4f3da6 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:02.408453 30200 leader_election.cc:302] T d7ef72cb49c64085a6d5b04228e275e4 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, 8a666e798fe8477d8dd1e51b385d5ead; no voters: fbbc2be6382e47908ecb7c68c367bd26
I20240503 02:13:02.409154 1453 raft_consensus.cc:2781] T d7ef72cb49c64085a6d5b04228e275e4 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:02.409304 1453 raft_consensus.cc:482] T d7ef72cb49c64085a6d5b04228e275e4 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:02.409408 1453 raft_consensus.cc:3037] T d7ef72cb49c64085a6d5b04228e275e4 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:02.416705 1453 raft_consensus.cc:504] T d7ef72cb49c64085a6d5b04228e275e4 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:02.418268 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7ef72cb49c64085a6d5b04228e275e4" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:13:02.418545 305 raft_consensus.cc:3037] T d7ef72cb49c64085a6d5b04228e275e4 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:02.421294 1352 raft_consensus.cc:686] T cf0743dee35b411e8fd069322d4f3da6 P 9a4d92bc25374fb3960fde7d10be309d [term 1 LEADER]: Becoming Leader. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Running, Role: LEADER
I20240503 02:13:02.421814 1352 consensus_queue.cc:237] T cf0743dee35b411e8fd069322d4f3da6 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:02.425299 1376 tablet_bootstrap.cc:492] T b6c9fd3cbbd5453897aa072600db569d P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:02.425786 1376 tablet_bootstrap.cc:492] T b6c9fd3cbbd5453897aa072600db569d P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap complete.
I20240503 02:13:02.426040 1376 ts_tablet_manager.cc:1367] T b6c9fd3cbbd5453897aa072600db569d P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.271s user 0.091s sys 0.061s
I20240503 02:13:02.426919 1376 raft_consensus.cc:348] T b6c9fd3cbbd5453897aa072600db569d P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:02.427141 1376 raft_consensus.cc:729] T b6c9fd3cbbd5453897aa072600db569d P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:02.427376 1376 consensus_queue.cc:260] T b6c9fd3cbbd5453897aa072600db569d P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:02.429101 1376 ts_tablet_manager.cc:1398] T b6c9fd3cbbd5453897aa072600db569d P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240503 02:13:02.429893 30156 tablet_copy_service.cc:342] P a9094240860e43bfb129193f3bfd5be6: Request end of tablet copy session fbbc2be6382e47908ecb7c68c367bd26-b6c9fd3cbbd5453897aa072600db569d received from {username='slave'} at 127.22.108.7:37671
I20240503 02:13:02.430052 30156 tablet_copy_service.cc:433] P a9094240860e43bfb129193f3bfd5be6: ending tablet copy session fbbc2be6382e47908ecb7c68c367bd26-b6c9fd3cbbd5453897aa072600db569d on tablet b6c9fd3cbbd5453897aa072600db569d with peer fbbc2be6382e47908ecb7c68c367bd26
I20240503 02:13:02.434028 29731 catalog_manager.cc:5484] T cf0743dee35b411e8fd069322d4f3da6 P 9a4d92bc25374fb3960fde7d10be309d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5). New cstate: current_term: 1 leader_uuid: "9a4d92bc25374fb3960fde7d10be309d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:02.435783 305 raft_consensus.cc:2454] T d7ef72cb49c64085a6d5b04228e275e4 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:13:02.436378 30200 leader_election.cc:302] T d7ef72cb49c64085a6d5b04228e275e4 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, 8a666e798fe8477d8dd1e51b385d5ead; no voters:
I20240503 02:13:02.436780 1180 raft_consensus.cc:2781] T d7ef72cb49c64085a6d5b04228e275e4 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:02.438961 1453 leader_election.cc:288] T d7ef72cb49c64085a6d5b04228e275e4 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Requested vote from peers 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:02.439596 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7ef72cb49c64085a6d5b04228e275e4" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
W20240503 02:13:02.440198 30200 leader_election.cc:341] T d7ef72cb49c64085a6d5b04228e275e4 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:02.454157 1512 tablet_bootstrap.cc:492] T 2baf70e0e1894cea9fdeb92279bc0cff P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:02.454470 1512 ts_tablet_manager.cc:1367] T 2baf70e0e1894cea9fdeb92279bc0cff P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.110s user 0.049s sys 0.023s
I20240503 02:13:02.455332 1512 raft_consensus.cc:348] T 2baf70e0e1894cea9fdeb92279bc0cff P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:02.455644 1512 raft_consensus.cc:374] T 2baf70e0e1894cea9fdeb92279bc0cff P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:02.455750 1512 raft_consensus.cc:729] T 2baf70e0e1894cea9fdeb92279bc0cff P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:02.456053 1512 consensus_queue.cc:260] T 2baf70e0e1894cea9fdeb92279bc0cff P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:02.457640 1512 ts_tablet_manager.cc:1398] T 2baf70e0e1894cea9fdeb92279bc0cff P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:13:02.457968 1512 tablet_bootstrap.cc:492] T 331905034afd48c19eb2bc2e2ec40280 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:02.460937 1512 tablet_bootstrap.cc:654] T 331905034afd48c19eb2bc2e2ec40280 P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:02.465974 1180 raft_consensus.cc:686] T d7ef72cb49c64085a6d5b04228e275e4 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Becoming Leader. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Running, Role: LEADER
I20240503 02:13:02.466475 1180 consensus_queue.cc:237] T d7ef72cb49c64085a6d5b04228e275e4 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:02.469154 1521 tablet_bootstrap.cc:492] T d7ef72cb49c64085a6d5b04228e275e4 P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:02.469360 1521 ts_tablet_manager.cc:1367] T d7ef72cb49c64085a6d5b04228e275e4 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.077s user 0.006s sys 0.000s
I20240503 02:13:02.470180 1521 raft_consensus.cc:348] T d7ef72cb49c64085a6d5b04228e275e4 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:02.473320 1521 raft_consensus.cc:374] T d7ef72cb49c64085a6d5b04228e275e4 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:02.473604 1521 raft_consensus.cc:729] T d7ef72cb49c64085a6d5b04228e275e4 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:02.473965 1521 consensus_queue.cc:260] T d7ef72cb49c64085a6d5b04228e275e4 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:02.476078 1521 ts_tablet_manager.cc:1398] T d7ef72cb49c64085a6d5b04228e275e4 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:13:02.488447 29731 catalog_manager.cc:5484] T d7ef72cb49c64085a6d5b04228e275e4 P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4). New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:02.498430 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf0743dee35b411e8fd069322d4f3da6" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:13:02.498672 30264 raft_consensus.cc:2454] T cf0743dee35b411e8fd069322d4f3da6 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 0.
I20240503 02:13:02.498965 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf0743dee35b411e8fd069322d4f3da6" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:13:02.499104 30266 raft_consensus.cc:3037] T cf0743dee35b411e8fd069322d4f3da6 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:02.501955 30266 raft_consensus.cc:2454] T cf0743dee35b411e8fd069322d4f3da6 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 1.
I20240503 02:13:02.506630 1512 tablet_bootstrap.cc:492] T 331905034afd48c19eb2bc2e2ec40280 P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:02.506798 1512 ts_tablet_manager.cc:1367] T 331905034afd48c19eb2bc2e2ec40280 P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.049s user 0.014s sys 0.024s
I20240503 02:13:02.507557 1512 raft_consensus.cc:348] T 331905034afd48c19eb2bc2e2ec40280 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:02.507763 1512 raft_consensus.cc:374] T 331905034afd48c19eb2bc2e2ec40280 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:02.507850 1512 raft_consensus.cc:729] T 331905034afd48c19eb2bc2e2ec40280 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:02.508092 1512 consensus_queue.cc:260] T 331905034afd48c19eb2bc2e2ec40280 P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:02.509744 1512 ts_tablet_manager.cc:1398] T 331905034afd48c19eb2bc2e2ec40280 P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:13:02.535955 436 raft_consensus.cc:1203] T b6c9fd3cbbd5453897aa072600db569d P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:02.583014 30136 consensus_queue.cc:237] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:02.586210 30006 raft_consensus.cc:1261] T 88b57085de3144cea3f64922ae5ccf55 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:02.586274 30266 raft_consensus.cc:1261] T 88b57085de3144cea3f64922ae5ccf55 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:02.587081 912 consensus_queue.cc:1035] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:02.587451 1346 consensus_queue.cc:1035] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:02.591004 912 raft_consensus.cc:2932] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:02.591598 30006 raft_consensus.cc:2932] T 88b57085de3144cea3f64922ae5ccf55 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:02.596755 29731 catalog_manager.cc:5484] T 88b57085de3144cea3f64922ae5ccf55 P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: config changed from index -1 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:02.597318 30266 raft_consensus.cc:2932] T 88b57085de3144cea3f64922ae5ccf55 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
W20240503 02:13:02.599391 30069 consensus_peers.cc:485] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 88b57085de3144cea3f64922ae5ccf55. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:02.610874 1183 raft_consensus.cc:482] T 331905034afd48c19eb2bc2e2ec40280 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:02.611049 1183 raft_consensus.cc:504] T 331905034afd48c19eb2bc2e2ec40280 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:02.611718 1183 leader_election.cc:288] T 331905034afd48c19eb2bc2e2ec40280 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:13:02.612504 1181 raft_consensus.cc:482] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:02.612529 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331905034afd48c19eb2bc2e2ec40280" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:13:02.612509 30006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331905034afd48c19eb2bc2e2ec40280" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:13:02.612749 32640 raft_consensus.cc:2454] T 331905034afd48c19eb2bc2e2ec40280 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:13:02.612668 1181 raft_consensus.cc:504] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:02.612761 30006 raft_consensus.cc:2454] T 331905034afd48c19eb2bc2e2ec40280 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:13:02.613440 1181 leader_election.cc:288] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:13:02.613971 32709 leader_election.cc:302] T 331905034afd48c19eb2bc2e2ec40280 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 8a666e798fe8477d8dd1e51b385d5ead; no voters:
I20240503 02:13:02.614256 1183 raft_consensus.cc:2781] T 331905034afd48c19eb2bc2e2ec40280 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:02.614372 1183 raft_consensus.cc:482] T 331905034afd48c19eb2bc2e2ec40280 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:02.614462 1183 raft_consensus.cc:3037] T 331905034afd48c19eb2bc2e2ec40280 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:02.615444 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2baf70e0e1894cea9fdeb92279bc0cff" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:13:02.615643 305 raft_consensus.cc:2454] T 2baf70e0e1894cea9fdeb92279bc0cff P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 0.
I20240503 02:13:02.616094 371 leader_election.cc:302] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:02.616525 1181 raft_consensus.cc:2781] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:02.616627 1181 raft_consensus.cc:482] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:02.616698 1181 raft_consensus.cc:3037] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:02.617223 1183 raft_consensus.cc:504] T 331905034afd48c19eb2bc2e2ec40280 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:02.617805 1183 leader_election.cc:288] T 331905034afd48c19eb2bc2e2ec40280 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Requested vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:13:02.619051 1181 raft_consensus.cc:504] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:02.619635 1181 leader_election.cc:288] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Requested vote from peers 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:13:02.620311 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2baf70e0e1894cea9fdeb92279bc0cff" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:13:02.620577 32640 raft_consensus.cc:2454] T 2baf70e0e1894cea9fdeb92279bc0cff P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 0.
I20240503 02:13:02.621445 30006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331905034afd48c19eb2bc2e2ec40280" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:13:02.621691 30006 raft_consensus.cc:3037] T 331905034afd48c19eb2bc2e2ec40280 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:02.621837 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331905034afd48c19eb2bc2e2ec40280" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d"
I20240503 02:13:02.622020 32640 raft_consensus.cc:3037] T 331905034afd48c19eb2bc2e2ec40280 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:02.624521 30006 raft_consensus.cc:2454] T 331905034afd48c19eb2bc2e2ec40280 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:13:02.624670 32640 raft_consensus.cc:2454] T 331905034afd48c19eb2bc2e2ec40280 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:13:02.625121 32709 leader_election.cc:302] T 331905034afd48c19eb2bc2e2ec40280 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, 9a4d92bc25374fb3960fde7d10be309d; no voters:
I20240503 02:13:02.625649 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2baf70e0e1894cea9fdeb92279bc0cff" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:13:02.625835 305 raft_consensus.cc:3037] T 2baf70e0e1894cea9fdeb92279bc0cff P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:02.625849 1183 raft_consensus.cc:2781] T 331905034afd48c19eb2bc2e2ec40280 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:02.626049 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2baf70e0e1894cea9fdeb92279bc0cff" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d"
I20240503 02:13:02.628093 32640 raft_consensus.cc:3037] T 2baf70e0e1894cea9fdeb92279bc0cff P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:02.628583 305 raft_consensus.cc:2454] T 2baf70e0e1894cea9fdeb92279bc0cff P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 1.
I20240503 02:13:02.629096 371 leader_election.cc:302] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:02.629426 1568 raft_consensus.cc:2781] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:02.629756 1568 raft_consensus.cc:686] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Becoming Leader. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Running, Role: LEADER
I20240503 02:13:02.630136 1568 consensus_queue.cc:237] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:02.632524 32640 raft_consensus.cc:2454] T 2baf70e0e1894cea9fdeb92279bc0cff P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 1.
I20240503 02:13:02.636632 1183 raft_consensus.cc:686] T 331905034afd48c19eb2bc2e2ec40280 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Becoming Leader. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Running, Role: LEADER
I20240503 02:13:02.637064 1183 consensus_queue.cc:237] T 331905034afd48c19eb2bc2e2ec40280 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:02.638401 29725 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:13:02.641597 29731 catalog_manager.cc:5484] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7). New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:02.646096 29730 catalog_manager.cc:5484] T 331905034afd48c19eb2bc2e2ec40280 P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6). New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:02.657207 436 raft_consensus.cc:3037] T d7ef72cb49c64085a6d5b04228e275e4 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:02.657439 305 raft_consensus.cc:1261] T d7ef72cb49c64085a6d5b04228e275e4 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:02.658000 1180 consensus_queue.cc:1035] T d7ef72cb49c64085a6d5b04228e275e4 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:02.660241 436 raft_consensus.cc:1261] T d7ef72cb49c64085a6d5b04228e275e4 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:02.660831 1180 consensus_queue.cc:1035] T d7ef72cb49c64085a6d5b04228e275e4 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:02.662448 1580 tablet.cc:1722] T d7ef72cb49c64085a6d5b04228e275e4 P 1f10f80f13c84e80b03eda7085a3b012: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:02.663436 1580 tablet.cc:1620] T d7ef72cb49c64085a6d5b04228e275e4 P 1f10f80f13c84e80b03eda7085a3b012: MemRowSet was empty: no flush needed.
I20240503 02:13:02.663774 29730 catalog_manager.cc:5576] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975) does not have the latest schema for tablet d7ef72cb49c64085a6d5b04228e275e4 (table rebalancer_dynamic_table [id=42b5416694924d77976fd377a8418575]). Expected version 1 got 0
I20240503 02:13:02.665239 1582 tablet.cc:1722] T d7ef72cb49c64085a6d5b04228e275e4 P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:02.665859 1582 tablet.cc:1620] T d7ef72cb49c64085a6d5b04228e275e4 P 8a666e798fe8477d8dd1e51b385d5ead: MemRowSet was empty: no flush needed.
I20240503 02:13:02.674216 29731 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=42b5416694924d77976fd377a8418575] alter complete (version 1)
I20240503 02:13:02.675845 568 consensus_queue.cc:237] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:02.679064 305 raft_consensus.cc:1261] T 6bd4fcc4faf44d148a03beaafc1c05ba P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:13:02.680194 502 consensus_peers.cc:485] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6bd4fcc4faf44d148a03beaafc1c05ba. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:02.680207 29872 raft_consensus.cc:1261] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:02.681370 1372 consensus_queue.cc:1035] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:02.681802 1584 consensus_queue.cc:1035] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:02.691219 29731 catalog_manager.cc:5576] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241) does not have the latest schema for tablet d7ef72cb49c64085a6d5b04228e275e4 (table rebalancer_dynamic_table [id=42b5416694924d77976fd377a8418575]). Expected version 1 got 0
I20240503 02:13:02.691447 1589 tablet.cc:1722] T d7ef72cb49c64085a6d5b04228e275e4 P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:02.692104 1589 tablet.cc:1620] T d7ef72cb49c64085a6d5b04228e275e4 P fbbc2be6382e47908ecb7c68c367bd26: MemRowSet was empty: no flush needed.
I20240503 02:13:02.698365 1584 raft_consensus.cc:2932] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:02.699234 305 raft_consensus.cc:2932] T 6bd4fcc4faf44d148a03beaafc1c05ba P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:02.703644 29872 raft_consensus.cc:2932] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:02.706749 29731 catalog_manager.cc:5484] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: config changed from index -1 to 2, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New cstate: current_term: 1 leader_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:02.739509 29872 consensus_queue.cc:237] T df37d88053e2419fbe40d03023d0c756 P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:02.741869 30266 raft_consensus.cc:1261] T df37d88053e2419fbe40d03023d0c756 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:13:02.742480 29809 consensus_peers.cc:485] T df37d88053e2419fbe40d03023d0c756 P 16440e365430405fb10366ab70e2b429 -> Peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): Couldn't send request to peer fbbc2be6382e47908ecb7c68c367bd26. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df37d88053e2419fbe40d03023d0c756. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:02.743383 1607 consensus_queue.cc:1035] T df37d88053e2419fbe40d03023d0c756 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:02.743764 30136 raft_consensus.cc:1261] T df37d88053e2419fbe40d03023d0c756 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:02.744211 1607 consensus_queue.cc:1035] T df37d88053e2419fbe40d03023d0c756 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:02.746855 1280 raft_consensus.cc:2932] T df37d88053e2419fbe40d03023d0c756 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:02.747519 30136 raft_consensus.cc:2932] T df37d88053e2419fbe40d03023d0c756 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:02.750950 30266 raft_consensus.cc:2932] T df37d88053e2419fbe40d03023d0c756 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:02.754770 29730 catalog_manager.cc:5484] T df37d88053e2419fbe40d03023d0c756 P 16440e365430405fb10366ab70e2b429 reported cstate change: config changed from index -1 to 4, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:02.778779 29731 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:13:02.788378 1580 tablet.cc:1722] T d7ef72cb49c64085a6d5b04228e275e4 P 1f10f80f13c84e80b03eda7085a3b012: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:02.788843 1580 tablet.cc:1620] T d7ef72cb49c64085a6d5b04228e275e4 P 1f10f80f13c84e80b03eda7085a3b012: MemRowSet was empty: no flush needed.
I20240503 02:13:02.789057 1582 tablet.cc:1722] T d7ef72cb49c64085a6d5b04228e275e4 P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:02.789510 1582 tablet.cc:1620] T d7ef72cb49c64085a6d5b04228e275e4 P 8a666e798fe8477d8dd1e51b385d5ead: MemRowSet was empty: no flush needed.
I20240503 02:13:02.789907 1589 tablet.cc:1722] T d7ef72cb49c64085a6d5b04228e275e4 P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:02.790331 1589 tablet.cc:1620] T d7ef72cb49c64085a6d5b04228e275e4 P fbbc2be6382e47908ecb7c68c367bd26: MemRowSet was empty: no flush needed.
I20240503 02:13:02.793679 30136 consensus_queue.cc:237] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:02.797731 29725 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=42b5416694924d77976fd377a8418575] alter complete (version 2)
I20240503 02:13:02.801888 29872 raft_consensus.cc:1261] T 779a6685fc8c4670a71b51c4493bc2f7 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:02.802623 912 consensus_queue.cc:1035] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:02.803011 30266 raft_consensus.cc:1261] T 779a6685fc8c4670a71b51c4493bc2f7 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:02.803491 912 consensus_queue.cc:1035] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:13:02.809950 30069 consensus_peers.cc:485] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 779a6685fc8c4670a71b51c4493bc2f7. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:02.810446 912 raft_consensus.cc:2932] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:02.811003 30266 raft_consensus.cc:2932] T 779a6685fc8c4670a71b51c4493bc2f7 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:02.816498 29725 catalog_manager.cc:5484] T 779a6685fc8c4670a71b51c4493bc2f7 P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: config changed from index -1 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:02.820487 29872 raft_consensus.cc:2932] T 779a6685fc8c4670a71b51c4493bc2f7 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:02.902429 29725 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:13:02.913002 1580 tablet.cc:1722] T d7ef72cb49c64085a6d5b04228e275e4 P 1f10f80f13c84e80b03eda7085a3b012: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:02.916385 1582 tablet.cc:1722] T d7ef72cb49c64085a6d5b04228e275e4 P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:02.923089 1589 tablet.cc:1722] T d7ef72cb49c64085a6d5b04228e275e4 P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:02.929006 29725 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table_new_name [id=42b5416694924d77976fd377a8418575] alter complete (version 3)
I20240503 02:13:02.948544 1352 consensus_queue.cc:1035] T cf0743dee35b411e8fd069322d4f3da6 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:02.948753 30264 consensus_queue.cc:237] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:02.951521 30136 raft_consensus.cc:1261] T 6bf84757d1fb44689da830a82e857787 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:02.952283 30006 raft_consensus.cc:1261] T 6bf84757d1fb44689da830a82e857787 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:13:02.952767 30200 consensus_peers.cc:485] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6bf84757d1fb44689da830a82e857787. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:02.952940 1533 consensus_queue.cc:1035] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:02.953294 1180 consensus_queue.cc:1035] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:02.962447 1180 raft_consensus.cc:2932] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:02.963063 1650 consensus_queue.cc:1035] T cf0743dee35b411e8fd069322d4f3da6 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:02.963096 30136 raft_consensus.cc:2932] T 6bf84757d1fb44689da830a82e857787 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:02.967669 30006 raft_consensus.cc:2932] T 6bf84757d1fb44689da830a82e857787 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:02.972680 29730 catalog_manager.cc:5484] T 6bf84757d1fb44689da830a82e857787 P a9094240860e43bfb129193f3bfd5be6 reported cstate change: config changed from index -1 to 4, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:03.003079 29872 consensus_queue.cc:237] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:03.005326 30264 raft_consensus.cc:1261] T 32a2dbf80daf40d6b21442a39f62f14c P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:03.005353 30136 raft_consensus.cc:1261] T 32a2dbf80daf40d6b21442a39f62f14c P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:13:03.005489 29809 consensus_peers.cc:485] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 -> Peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): Couldn't send request to peer 8a666e798fe8477d8dd1e51b385d5ead. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32a2dbf80daf40d6b21442a39f62f14c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:03.005954 1484 consensus_queue.cc:1035] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:03.006433 1280 consensus_queue.cc:1035] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:03.008904 1484 raft_consensus.cc:2932] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:03.009464 30136 raft_consensus.cc:2932] T 32a2dbf80daf40d6b21442a39f62f14c P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:03.015195 29734 catalog_manager.cc:5484] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 reported cstate change: config changed from index -1 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:03.015862 30264 raft_consensus.cc:2932] T 32a2dbf80daf40d6b21442a39f62f14c P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:03.033828 29725 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:03.034068 29725 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:03.037148 29725 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet d7ef72cb49c64085a6d5b04228e275e4
I20240503 02:13:03.038076 32755 tablet_service.cc:1514] Processing DeleteTablet for tablet d7ef72cb49c64085a6d5b04228e275e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:03 UTC) from {username='slave'} at 127.0.0.1:46360
I20240503 02:13:03.038164 30245 tablet_service.cc:1514] Processing DeleteTablet for tablet d7ef72cb49c64085a6d5b04228e275e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:03 UTC) from {username='slave'} at 127.0.0.1:49856
I20240503 02:13:03.038481 417 tablet_service.cc:1514] Processing DeleteTablet for tablet d7ef72cb49c64085a6d5b04228e275e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:03 UTC) from {username='slave'} at 127.0.0.1:50814
I20240503 02:13:03.038878 1682 tablet_replica.cc:330] T d7ef72cb49c64085a6d5b04228e275e4 P 1f10f80f13c84e80b03eda7085a3b012: stopping tablet replica
I20240503 02:13:03.039050 1683 tablet_replica.cc:330] T d7ef72cb49c64085a6d5b04228e275e4 P fbbc2be6382e47908ecb7c68c367bd26: stopping tablet replica
I20240503 02:13:03.039096 1682 raft_consensus.cc:2229] T d7ef72cb49c64085a6d5b04228e275e4 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:03.039252 1683 raft_consensus.cc:2229] T d7ef72cb49c64085a6d5b04228e275e4 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:03.039470 1683 raft_consensus.cc:2258] T d7ef72cb49c64085a6d5b04228e275e4 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:03.039551 1682 raft_consensus.cc:2258] T d7ef72cb49c64085a6d5b04228e275e4 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:03.040791 1682 ts_tablet_manager.cc:1875] T d7ef72cb49c64085a6d5b04228e275e4 P 1f10f80f13c84e80b03eda7085a3b012: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:03.041152 1683 ts_tablet_manager.cc:1875] T d7ef72cb49c64085a6d5b04228e275e4 P fbbc2be6382e47908ecb7c68c367bd26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:03.048988 1683 ts_tablet_manager.cc:1888] T d7ef72cb49c64085a6d5b04228e275e4 P fbbc2be6382e47908ecb7c68c367bd26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:03.049283 1683 log.cc:1196] T d7ef72cb49c64085a6d5b04228e275e4 P fbbc2be6382e47908ecb7c68c367bd26: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal/wals/d7ef72cb49c64085a6d5b04228e275e4
I20240503 02:13:03.049916 1683 ts_tablet_manager.cc:1909] T d7ef72cb49c64085a6d5b04228e275e4 P fbbc2be6382e47908ecb7c68c367bd26: Deleting consensus metadata
I20240503 02:13:03.051410 1681 tablet_replica.cc:330] T d7ef72cb49c64085a6d5b04228e275e4 P 8a666e798fe8477d8dd1e51b385d5ead: stopping tablet replica
I20240503 02:13:03.051385 29719 catalog_manager.cc:4830] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): tablet d7ef72cb49c64085a6d5b04228e275e4 (table rebalancer_dynamic_table_new_name [id=42b5416694924d77976fd377a8418575]) successfully deleted
I20240503 02:13:03.051748 1681 raft_consensus.cc:2229] T d7ef72cb49c64085a6d5b04228e275e4 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:03.052052 1681 raft_consensus.cc:2258] T d7ef72cb49c64085a6d5b04228e275e4 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:03.053205 1681 ts_tablet_manager.cc:1875] T d7ef72cb49c64085a6d5b04228e275e4 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:03.053704 1682 ts_tablet_manager.cc:1888] T d7ef72cb49c64085a6d5b04228e275e4 P 1f10f80f13c84e80b03eda7085a3b012: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:03.053880 1682 log.cc:1196] T d7ef72cb49c64085a6d5b04228e275e4 P 1f10f80f13c84e80b03eda7085a3b012: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/wals/d7ef72cb49c64085a6d5b04228e275e4
I20240503 02:13:03.054358 1682 ts_tablet_manager.cc:1909] T d7ef72cb49c64085a6d5b04228e275e4 P 1f10f80f13c84e80b03eda7085a3b012: Deleting consensus metadata
I20240503 02:13:03.055411 29719 catalog_manager.cc:4830] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): tablet d7ef72cb49c64085a6d5b04228e275e4 (table rebalancer_dynamic_table_new_name [id=42b5416694924d77976fd377a8418575]) successfully deleted
I20240503 02:13:03.058519 30006 consensus_queue.cc:237] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:03.058840 1681 ts_tablet_manager.cc:1888] T d7ef72cb49c64085a6d5b04228e275e4 P 8a666e798fe8477d8dd1e51b385d5ead: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:03.058956 1681 log.cc:1196] T d7ef72cb49c64085a6d5b04228e275e4 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal/wals/d7ef72cb49c64085a6d5b04228e275e4
I20240503 02:13:03.059473 1681 ts_tablet_manager.cc:1909] T d7ef72cb49c64085a6d5b04228e275e4 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting consensus metadata
I20240503 02:13:03.060532 29719 catalog_manager.cc:4830] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): tablet d7ef72cb49c64085a6d5b04228e275e4 (table rebalancer_dynamic_table_new_name [id=42b5416694924d77976fd377a8418575]) successfully deleted
I20240503 02:13:03.061093 29872 raft_consensus.cc:1261] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:03.061688 1428 consensus_queue.cc:1035] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:13:03.062027 29940 consensus_peers.cc:485] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 155735c7f8b14997ac4c770aee2a9401 -> Peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): Couldn't send request to peer fbbc2be6382e47908ecb7c68c367bd26. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e61579e74a1a4a9dbd9a09f9b34d9c8d. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:03.062083 30136 raft_consensus.cc:1261] T e61579e74a1a4a9dbd9a09f9b34d9c8d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:03.062608 1428 consensus_queue.cc:1035] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:03.065312 1690 raft_consensus.cc:2932] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:03.065901 30134 raft_consensus.cc:2932] T e61579e74a1a4a9dbd9a09f9b34d9c8d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:03.070420 29872 raft_consensus.cc:2932] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:03.074304 29734 catalog_manager.cc:5484] T e61579e74a1a4a9dbd9a09f9b34d9c8d P a9094240860e43bfb129193f3bfd5be6 reported cstate change: config changed from index -1 to 4, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:03.076849 1694 ts_tablet_manager.cc:902] T 88b57085de3144cea3f64922ae5ccf55 P f575ce92c0224eeb80b9fae5f6561fbe: Initiating tablet copy from peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:13:03.078307 1183 consensus_queue.cc:1035] T 331905034afd48c19eb2bc2e2ec40280 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:03.093605 1183 consensus_queue.cc:1035] T 331905034afd48c19eb2bc2e2ec40280 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:03.111001 1694 tablet_copy_client.cc:308] T 88b57085de3144cea3f64922ae5ccf55 P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.3:42679
I20240503 02:13:03.121227 1708 ts_tablet_manager.cc:902] T 6bd4fcc4faf44d148a03beaafc1c05ba P 9a4d92bc25374fb3960fde7d10be309d: Initiating tablet copy from peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:13:03.122978 30156 tablet_copy_service.cc:140] P a9094240860e43bfb129193f3bfd5be6: Received BeginTabletCopySession request for tablet 88b57085de3144cea3f64922ae5ccf55 from peer f575ce92c0224eeb80b9fae5f6561fbe ({username='slave'} at 127.22.108.8:37273)
I20240503 02:13:03.123193 30156 tablet_copy_service.cc:161] P a9094240860e43bfb129193f3bfd5be6: Beginning new tablet copy session on tablet 88b57085de3144cea3f64922ae5ccf55 from peer f575ce92c0224eeb80b9fae5f6561fbe at {username='slave'} at 127.22.108.8:37273: session id = f575ce92c0224eeb80b9fae5f6561fbe-88b57085de3144cea3f64922ae5ccf55
I20240503 02:13:03.124481 1708 tablet_copy_client.cc:308] T 6bd4fcc4faf44d148a03beaafc1c05ba P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.8:42233
I20240503 02:13:03.125399 588 tablet_copy_service.cc:140] P f575ce92c0224eeb80b9fae5f6561fbe: Received BeginTabletCopySession request for tablet 6bd4fcc4faf44d148a03beaafc1c05ba from peer 9a4d92bc25374fb3960fde7d10be309d ({username='slave'} at 127.22.108.5:33527)
I20240503 02:13:03.125612 588 tablet_copy_service.cc:161] P f575ce92c0224eeb80b9fae5f6561fbe: Beginning new tablet copy session on tablet 6bd4fcc4faf44d148a03beaafc1c05ba from peer 9a4d92bc25374fb3960fde7d10be309d at {username='slave'} at 127.22.108.5:33527: session id = 9a4d92bc25374fb3960fde7d10be309d-6bd4fcc4faf44d148a03beaafc1c05ba
I20240503 02:13:03.125797 30156 tablet_copy_source_session.cc:205] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:03.126977 1694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88b57085de3144cea3f64922ae5ccf55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.129653 588 tablet_copy_source_session.cc:205] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:03.131601 1708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bd4fcc4faf44d148a03beaafc1c05ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.134466 1694 tablet_copy_client.cc:783] T 88b57085de3144cea3f64922ae5ccf55 P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:03.134718 1694 tablet_copy_client.cc:647] T 88b57085de3144cea3f64922ae5ccf55 P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:03.137228 1694 tablet_copy_client.cc:523] T 88b57085de3144cea3f64922ae5ccf55 P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:03.139497 29734 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:13:03.141000 1694 tablet_bootstrap.cc:492] T 88b57085de3144cea3f64922ae5ccf55 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:03.143010 1708 tablet_copy_client.cc:783] T 6bd4fcc4faf44d148a03beaafc1c05ba P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:03.143255 1708 tablet_copy_client.cc:647] T 6bd4fcc4faf44d148a03beaafc1c05ba P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:03.145190 1568 consensus_queue.cc:1035] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:03.150185 32755 tablet_service.cc:1467] Processing CreateTablet for tablet 4ab3b073d9f845cdb76a8b4aa0929924 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=99a9a7da96cb474a8c0a1a83c7fbdb4b]), partition=
I20240503 02:13:03.150301 417 tablet_service.cc:1467] Processing CreateTablet for tablet 4ab3b073d9f845cdb76a8b4aa0929924 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=99a9a7da96cb474a8c0a1a83c7fbdb4b]), partition=
I20240503 02:13:03.150597 32755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ab3b073d9f845cdb76a8b4aa0929924. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.150674 417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ab3b073d9f845cdb76a8b4aa0929924. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.153920 32620 tablet_service.cc:1467] Processing CreateTablet for tablet 4ab3b073d9f845cdb76a8b4aa0929924 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=99a9a7da96cb474a8c0a1a83c7fbdb4b]), partition=
I20240503 02:13:03.154294 32620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ab3b073d9f845cdb76a8b4aa0929924. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.160259 1708 tablet_copy_client.cc:523] T 6bd4fcc4faf44d148a03beaafc1c05ba P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:03.170105 1181 consensus_queue.cc:1035] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:03.172677 1708 tablet_bootstrap.cc:492] T 6bd4fcc4faf44d148a03beaafc1c05ba P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:03.205214 1724 tablet_bootstrap.cc:492] T 4ab3b073d9f845cdb76a8b4aa0929924 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:03.205806 1721 tablet_bootstrap.cc:492] T 4ab3b073d9f845cdb76a8b4aa0929924 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:03.207587 29734 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_extra_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:03.209311 1721 tablet_bootstrap.cc:654] T 4ab3b073d9f845cdb76a8b4aa0929924 P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:03.210489 1726 tablet_bootstrap.cc:492] T 4ab3b073d9f845cdb76a8b4aa0929924 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:03.212270 1726 tablet_bootstrap.cc:654] T 4ab3b073d9f845cdb76a8b4aa0929924 P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:03.214901 1724 tablet_bootstrap.cc:654] T 4ab3b073d9f845cdb76a8b4aa0929924 P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:03.216513 1694 tablet_bootstrap.cc:492] T 88b57085de3144cea3f64922ae5ccf55 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:03.217106 1694 tablet_bootstrap.cc:492] T 88b57085de3144cea3f64922ae5ccf55 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap complete.
I20240503 02:13:03.217411 1694 ts_tablet_manager.cc:1367] T 88b57085de3144cea3f64922ae5ccf55 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.077s user 0.055s sys 0.012s
I20240503 02:13:03.218273 1694 raft_consensus.cc:348] T 88b57085de3144cea3f64922ae5ccf55 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:03.218538 1694 raft_consensus.cc:729] T 88b57085de3144cea3f64922ae5ccf55 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:03.218843 1694 consensus_queue.cc:260] T 88b57085de3144cea3f64922ae5ccf55 P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:03.219588 547 tablet_service.cc:1467] Processing CreateTablet for tablet e4fe8ae88ae843609a66c7a2e6c7b4af (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=098275b181bb4441990148dfd09ded67]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:03.220009 547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4fe8ae88ae843609a66c7a2e6c7b4af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.220522 1694 ts_tablet_manager.cc:1398] T 88b57085de3144cea3f64922ae5ccf55 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:13:03.221215 32752 tablet_service.cc:1467] Processing CreateTablet for tablet ac4a753bca5f44058199942f05ef181d (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=098275b181bb4441990148dfd09ded67]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:03.221223 32755 tablet_service.cc:1467] Processing CreateTablet for tablet e4fe8ae88ae843609a66c7a2e6c7b4af (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=098275b181bb4441990148dfd09ded67]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:03.221716 32752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac4a753bca5f44058199942f05ef181d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.221935 32755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4fe8ae88ae843609a66c7a2e6c7b4af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.222208 417 tablet_service.cc:1467] Processing CreateTablet for tablet 28f3f05f0a344cd683cb9e010273f908 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=098275b181bb4441990148dfd09ded67]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:03.222627 417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28f3f05f0a344cd683cb9e010273f908. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.224697 32620 tablet_service.cc:1467] Processing CreateTablet for tablet 28f3f05f0a344cd683cb9e010273f908 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=098275b181bb4441990148dfd09ded67]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:03.225129 32620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28f3f05f0a344cd683cb9e010273f908. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.225845 30245 tablet_service.cc:1467] Processing CreateTablet for tablet ac4a753bca5f44058199942f05ef181d (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=098275b181bb4441990148dfd09ded67]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:03.226305 30245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac4a753bca5f44058199942f05ef181d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.232295 30114 tablet_service.cc:1467] Processing CreateTablet for tablet e4fe8ae88ae843609a66c7a2e6c7b4af (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=098275b181bb4441990148dfd09ded67]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:03.232590 30116 tablet_service.cc:1467] Processing CreateTablet for tablet 28f3f05f0a344cd683cb9e010273f908 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=098275b181bb4441990148dfd09ded67]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:03.232749 30114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4fe8ae88ae843609a66c7a2e6c7b4af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.232998 30116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28f3f05f0a344cd683cb9e010273f908. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.239373 546 tablet_service.cc:1467] Processing CreateTablet for tablet ac4a753bca5f44058199942f05ef181d (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=098275b181bb4441990148dfd09ded67]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:03.239847 546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac4a753bca5f44058199942f05ef181d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.244302 30156 tablet_copy_service.cc:342] P a9094240860e43bfb129193f3bfd5be6: Request end of tablet copy session f575ce92c0224eeb80b9fae5f6561fbe-88b57085de3144cea3f64922ae5ccf55 received from {username='slave'} at 127.22.108.8:37273
I20240503 02:13:03.244438 30156 tablet_copy_service.cc:433] P a9094240860e43bfb129193f3bfd5be6: ending tablet copy session f575ce92c0224eeb80b9fae5f6561fbe-88b57085de3144cea3f64922ae5ccf55 on tablet 88b57085de3144cea3f64922ae5ccf55 with peer f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:13:03.247349 1733 ts_tablet_manager.cc:902] T 779a6685fc8c4670a71b51c4493bc2f7 P f575ce92c0224eeb80b9fae5f6561fbe: Initiating tablet copy from peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:13:03.247974 30264 consensus_queue.cc:237] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:03.254308 30136 raft_consensus.cc:1261] T 1c9200fa969a46e6a48c64293801382b P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:03.254906 29872 raft_consensus.cc:1261] T 1c9200fa969a46e6a48c64293801382b P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:03.254952 1453 consensus_queue.cc:1035] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:03.256100 1453 consensus_queue.cc:1035] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:13:03.256263 30199 consensus_peers.cc:485] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c9200fa969a46e6a48c64293801382b. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:03.266321 1733 tablet_copy_client.cc:308] T 779a6685fc8c4670a71b51c4493bc2f7 P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.3:42679
I20240503 02:13:03.267035 30156 tablet_copy_service.cc:140] P a9094240860e43bfb129193f3bfd5be6: Received BeginTabletCopySession request for tablet 779a6685fc8c4670a71b51c4493bc2f7 from peer f575ce92c0224eeb80b9fae5f6561fbe ({username='slave'} at 127.22.108.8:37273)
I20240503 02:13:03.267190 30156 tablet_copy_service.cc:161] P a9094240860e43bfb129193f3bfd5be6: Beginning new tablet copy session on tablet 779a6685fc8c4670a71b51c4493bc2f7 from peer f575ce92c0224eeb80b9fae5f6561fbe at {username='slave'} at 127.22.108.8:37273: session id = f575ce92c0224eeb80b9fae5f6561fbe-779a6685fc8c4670a71b51c4493bc2f7
I20240503 02:13:03.269556 30156 tablet_copy_source_session.cc:205] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:03.271271 1733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 779a6685fc8c4670a71b51c4493bc2f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.271762 1708 tablet_bootstrap.cc:492] T 6bd4fcc4faf44d148a03beaafc1c05ba P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:03.272328 1708 tablet_bootstrap.cc:492] T 6bd4fcc4faf44d148a03beaafc1c05ba P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap complete.
I20240503 02:13:03.272655 1708 ts_tablet_manager.cc:1367] T 6bd4fcc4faf44d148a03beaafc1c05ba P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.100s user 0.003s sys 0.010s
I20240503 02:13:03.274030 1708 raft_consensus.cc:348] T 6bd4fcc4faf44d148a03beaafc1c05ba P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:03.274366 1708 raft_consensus.cc:729] T 6bd4fcc4faf44d148a03beaafc1c05ba P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:03.275789 1708 consensus_queue.cc:260] T 6bd4fcc4faf44d148a03beaafc1c05ba P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:03.277431 1708 ts_tablet_manager.cc:1398] T 6bd4fcc4faf44d148a03beaafc1c05ba P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:13:03.278054 1733 tablet_copy_client.cc:783] T 779a6685fc8c4670a71b51c4493bc2f7 P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:03.279361 588 tablet_copy_service.cc:342] P f575ce92c0224eeb80b9fae5f6561fbe: Request end of tablet copy session 9a4d92bc25374fb3960fde7d10be309d-6bd4fcc4faf44d148a03beaafc1c05ba received from {username='slave'} at 127.22.108.5:33527
I20240503 02:13:03.279438 1733 tablet_copy_client.cc:647] T 779a6685fc8c4670a71b51c4493bc2f7 P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:03.279564 588 tablet_copy_service.cc:433] P f575ce92c0224eeb80b9fae5f6561fbe: ending tablet copy session 9a4d92bc25374fb3960fde7d10be309d-6bd4fcc4faf44d148a03beaafc1c05ba on tablet 6bd4fcc4faf44d148a03beaafc1c05ba with peer 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:13:03.280977 1743 tablet_bootstrap.cc:492] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:13:03.282505 1733 tablet_copy_client.cc:523] T 779a6685fc8c4670a71b51c4493bc2f7 P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:03.283779 1742 tablet_bootstrap.cc:492] T e4fe8ae88ae843609a66c7a2e6c7b4af P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:03.287101 1733 tablet_bootstrap.cc:492] T 779a6685fc8c4670a71b51c4493bc2f7 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:03.287103 1742 tablet_bootstrap.cc:654] T e4fe8ae88ae843609a66c7a2e6c7b4af P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:03.290544 1743 tablet_bootstrap.cc:654] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:03.291955 1741 tablet_bootstrap.cc:492] T ac4a753bca5f44058199942f05ef181d P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:13:03.294174 1741 tablet_bootstrap.cc:654] T ac4a753bca5f44058199942f05ef181d P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:03.300884 1742 tablet_bootstrap.cc:492] T e4fe8ae88ae843609a66c7a2e6c7b4af P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:03.301007 1726 tablet_bootstrap.cc:492] T 4ab3b073d9f845cdb76a8b4aa0929924 P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:03.301065 1742 ts_tablet_manager.cc:1367] T e4fe8ae88ae843609a66c7a2e6c7b4af P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.004s
I20240503 02:13:03.301303 1726 ts_tablet_manager.cc:1367] T 4ab3b073d9f845cdb76a8b4aa0929924 P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.091s user 0.007s sys 0.001s
I20240503 02:13:03.302846 1721 tablet_bootstrap.cc:492] T 4ab3b073d9f845cdb76a8b4aa0929924 P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:03.303064 1721 ts_tablet_manager.cc:1367] T 4ab3b073d9f845cdb76a8b4aa0929924 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.098s user 0.016s sys 0.016s
I20240503 02:13:03.303016 1726 raft_consensus.cc:348] T 4ab3b073d9f845cdb76a8b4aa0929924 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:03.303198 1743 tablet_bootstrap.cc:492] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:13:03.303375 1743 ts_tablet_manager.cc:1367] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.000s
I20240503 02:13:03.303313 1742 raft_consensus.cc:348] T e4fe8ae88ae843609a66c7a2e6c7b4af P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:03.303547 1742 raft_consensus.cc:374] T e4fe8ae88ae843609a66c7a2e6c7b4af P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:03.303589 1726 raft_consensus.cc:374] T 4ab3b073d9f845cdb76a8b4aa0929924 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:03.303655 1742 raft_consensus.cc:729] T e4fe8ae88ae843609a66c7a2e6c7b4af P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:03.303849 1726 raft_consensus.cc:729] T 4ab3b073d9f845cdb76a8b4aa0929924 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:03.303920 1742 consensus_queue.cc:260] T e4fe8ae88ae843609a66c7a2e6c7b4af P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:03.303948 1721 raft_consensus.cc:348] T 4ab3b073d9f845cdb76a8b4aa0929924 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:03.304499 1742 ts_tablet_manager.cc:1398] T e4fe8ae88ae843609a66c7a2e6c7b4af P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20240503 02:13:03.304416 1743 raft_consensus.cc:348] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:03.304620 1743 raft_consensus.cc:374] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:03.304713 1743 raft_consensus.cc:729] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:13:03.304961 1743 consensus_queue.cc:260] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:03.305166 1721 raft_consensus.cc:374] T 4ab3b073d9f845cdb76a8b4aa0929924 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:03.305294 1721 raft_consensus.cc:729] T 4ab3b073d9f845cdb76a8b4aa0929924 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:03.305526 1724 tablet_bootstrap.cc:492] T 4ab3b073d9f845cdb76a8b4aa0929924 P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:03.305550 1721 consensus_queue.cc:260] T 4ab3b073d9f845cdb76a8b4aa0929924 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:03.305748 1724 ts_tablet_manager.cc:1367] T 4ab3b073d9f845cdb76a8b4aa0929924 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.101s user 0.004s sys 0.004s
I20240503 02:13:03.306766 1743 ts_tablet_manager.cc:1398] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:03.306679 1724 raft_consensus.cc:348] T 4ab3b073d9f845cdb76a8b4aa0929924 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:03.306986 1724 raft_consensus.cc:374] T 4ab3b073d9f845cdb76a8b4aa0929924 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:03.307088 1743 tablet_bootstrap.cc:492] T 28f3f05f0a344cd683cb9e010273f908 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:13:03.307137 1724 raft_consensus.cc:729] T 4ab3b073d9f845cdb76a8b4aa0929924 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:03.307437 1724 consensus_queue.cc:260] T 4ab3b073d9f845cdb76a8b4aa0929924 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:03.307803 1726 consensus_queue.cc:260] T 4ab3b073d9f845cdb76a8b4aa0929924 P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:03.308763 1742 tablet_bootstrap.cc:492] T ac4a753bca5f44058199942f05ef181d P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:03.310039 1743 tablet_bootstrap.cc:654] T 28f3f05f0a344cd683cb9e010273f908 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:03.311247 1726 ts_tablet_manager.cc:1398] T 4ab3b073d9f845cdb76a8b4aa0929924 P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240503 02:13:03.309415 1724 ts_tablet_manager.cc:1398] T 4ab3b073d9f845cdb76a8b4aa0929924 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:03.313925 1724 tablet_bootstrap.cc:492] T 28f3f05f0a344cd683cb9e010273f908 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:03.318542 1726 tablet_bootstrap.cc:492] T 28f3f05f0a344cd683cb9e010273f908 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:03.322484 1721 ts_tablet_manager.cc:1398] T 4ab3b073d9f845cdb76a8b4aa0929924 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20240503 02:13:03.323029 1743 tablet_bootstrap.cc:492] T 28f3f05f0a344cd683cb9e010273f908 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:13:03.323065 1726 tablet_bootstrap.cc:654] T 28f3f05f0a344cd683cb9e010273f908 P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:03.323256 1743 ts_tablet_manager.cc:1367] T 28f3f05f0a344cd683cb9e010273f908 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.000s
I20240503 02:13:03.323592 1721 tablet_bootstrap.cc:492] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:03.326848 1741 tablet_bootstrap.cc:492] T ac4a753bca5f44058199942f05ef181d P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:13:03.327065 1741 ts_tablet_manager.cc:1367] T ac4a753bca5f44058199942f05ef181d P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.035s user 0.000s sys 0.004s
I20240503 02:13:03.327059 1743 raft_consensus.cc:348] T 28f3f05f0a344cd683cb9e010273f908 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:03.327697 1743 raft_consensus.cc:374] T 28f3f05f0a344cd683cb9e010273f908 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:03.327970 1743 raft_consensus.cc:729] T 28f3f05f0a344cd683cb9e010273f908 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:13:03.327912 1741 raft_consensus.cc:348] T ac4a753bca5f44058199942f05ef181d P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:03.328209 1741 raft_consensus.cc:374] T ac4a753bca5f44058199942f05ef181d P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:03.328338 1741 raft_consensus.cc:729] T ac4a753bca5f44058199942f05ef181d P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:13:03.328060 1180 raft_consensus.cc:2932] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:03.328375 1743 consensus_queue.cc:260] T 28f3f05f0a344cd683cb9e010273f908 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:03.328691 1741 consensus_queue.cc:260] T ac4a753bca5f44058199942f05ef181d P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:03.328967 1721 tablet_bootstrap.cc:654] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:03.330243 1741 ts_tablet_manager.cc:1398] T ac4a753bca5f44058199942f05ef181d P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:13:03.330310 1743 ts_tablet_manager.cc:1398] T 28f3f05f0a344cd683cb9e010273f908 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:13:03.330893 1724 tablet_bootstrap.cc:654] T 28f3f05f0a344cd683cb9e010273f908 P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:03.332638 30135 raft_consensus.cc:2932] T 1c9200fa969a46e6a48c64293801382b P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:03.331262 1742 tablet_bootstrap.cc:654] T ac4a753bca5f44058199942f05ef181d P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:03.340088 29872 raft_consensus.cc:2932] T 1c9200fa969a46e6a48c64293801382b P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:03.344162 1726 tablet_bootstrap.cc:492] T 28f3f05f0a344cd683cb9e010273f908 P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:03.344401 1726 ts_tablet_manager.cc:1367] T 28f3f05f0a344cd683cb9e010273f908 P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.003s
I20240503 02:13:03.343844 29729 catalog_manager.cc:5484] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: config changed from index -1 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:03.345782 1726 raft_consensus.cc:348] T 28f3f05f0a344cd683cb9e010273f908 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:03.346087 1726 raft_consensus.cc:374] T 28f3f05f0a344cd683cb9e010273f908 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:03.346211 1726 raft_consensus.cc:729] T 28f3f05f0a344cd683cb9e010273f908 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:03.346549 1726 consensus_queue.cc:260] T 28f3f05f0a344cd683cb9e010273f908 P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:03.347168 1726 ts_tablet_manager.cc:1398] T 28f3f05f0a344cd683cb9e010273f908 P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:03.348177 1724 tablet_bootstrap.cc:492] T 28f3f05f0a344cd683cb9e010273f908 P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:03.348400 1724 ts_tablet_manager.cc:1367] T 28f3f05f0a344cd683cb9e010273f908 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.006s
I20240503 02:13:03.350545 1724 raft_consensus.cc:348] T 28f3f05f0a344cd683cb9e010273f908 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:03.350761 1724 raft_consensus.cc:374] T 28f3f05f0a344cd683cb9e010273f908 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:03.350840 1724 raft_consensus.cc:729] T 28f3f05f0a344cd683cb9e010273f908 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:03.351155 1724 consensus_queue.cc:260] T 28f3f05f0a344cd683cb9e010273f908 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:03.352993 1724 ts_tablet_manager.cc:1398] T 28f3f05f0a344cd683cb9e010273f908 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:13:03.361346 1753 ts_tablet_manager.cc:902] T df37d88053e2419fbe40d03023d0c756 P fbbc2be6382e47908ecb7c68c367bd26: Initiating tablet copy from peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:13:03.362988 1533 raft_consensus.cc:482] T ac4a753bca5f44058199942f05ef181d P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:03.363237 1533 raft_consensus.cc:504] T ac4a753bca5f44058199942f05ef181d P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:03.363860 1708 ts_tablet_manager.cc:902] T 6bf84757d1fb44689da830a82e857787 P 9a4d92bc25374fb3960fde7d10be309d: Initiating tablet copy from peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:13:03.364156 1533 leader_election.cc:288] T ac4a753bca5f44058199942f05ef181d P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:03.366333 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4a753bca5f44058199942f05ef181d" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
W20240503 02:13:03.367172 30200 leader_election.cc:341] T ac4a753bca5f44058199942f05ef181d P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:03.367909 1628 raft_consensus.cc:482] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:03.368052 1628 raft_consensus.cc:504] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:03.368805 1628 leader_election.cc:288] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:03.369467 568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4a753bca5f44058199942f05ef181d" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:03.370046 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4fe8ae88ae843609a66c7a2e6c7b4af" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:03.370275 567 raft_consensus.cc:2454] T e4fe8ae88ae843609a66c7a2e6c7b4af P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 0.
W20240503 02:13:03.370705 30199 leader_election.cc:341] T ac4a753bca5f44058199942f05ef181d P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:03.370929 1721 tablet_bootstrap.cc:492] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:03.370882 30199 leader_election.cc:302] T ac4a753bca5f44058199942f05ef181d P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012; no voters: 8a666e798fe8477d8dd1e51b385d5ead, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:13:03.371129 1721 ts_tablet_manager.cc:1367] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.048s user 0.002s sys 0.002s
I20240503 02:13:03.371141 30069 leader_election.cc:302] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:03.371423 1628 raft_consensus.cc:2781] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:03.371529 1628 raft_consensus.cc:482] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:03.371608 1628 raft_consensus.cc:3037] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:03.371600 1533 raft_consensus.cc:2726] T ac4a753bca5f44058199942f05ef181d P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:03.372020 1721 raft_consensus.cc:348] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:03.372247 1721 raft_consensus.cc:374] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:03.372318 1721 raft_consensus.cc:729] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:03.372625 1721 consensus_queue.cc:260] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:03.373286 1721 ts_tablet_manager.cc:1398] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:13:03.373569 1721 tablet_bootstrap.cc:492] T e4fe8ae88ae843609a66c7a2e6c7b4af P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:03.374348 1628 raft_consensus.cc:504] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:03.376190 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4fe8ae88ae843609a66c7a2e6c7b4af" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:13:03.376420 567 raft_consensus.cc:3037] T e4fe8ae88ae843609a66c7a2e6c7b4af P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:03.379315 567 raft_consensus.cc:2454] T e4fe8ae88ae843609a66c7a2e6c7b4af P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 1.
I20240503 02:13:03.379761 30069 leader_election.cc:302] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:03.380071 1346 raft_consensus.cc:2781] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:03.383313 1628 leader_election.cc:288] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Requested vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:03.383497 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4fe8ae88ae843609a66c7a2e6c7b4af" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:13:03.384078 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4fe8ae88ae843609a66c7a2e6c7b4af" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:13:03.384786 1721 tablet_bootstrap.cc:654] T e4fe8ae88ae843609a66c7a2e6c7b4af P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
W20240503 02:13:03.384793 30070 leader_election.cc:341] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:13:03.385282 30070 leader_election.cc:341] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:03.392419 1346 raft_consensus.cc:686] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Becoming Leader. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Running, Role: LEADER
I20240503 02:13:03.392742 1346 consensus_queue.cc:237] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:03.403774 1721 tablet_bootstrap.cc:492] T e4fe8ae88ae843609a66c7a2e6c7b4af P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:03.404012 1721 ts_tablet_manager.cc:1367] T e4fe8ae88ae843609a66c7a2e6c7b4af P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.000s
I20240503 02:13:03.404871 1721 raft_consensus.cc:348] T e4fe8ae88ae843609a66c7a2e6c7b4af P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:03.405237 1721 raft_consensus.cc:374] T e4fe8ae88ae843609a66c7a2e6c7b4af P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:03.405404 1721 raft_consensus.cc:729] T e4fe8ae88ae843609a66c7a2e6c7b4af P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:03.405783 1721 consensus_queue.cc:260] T e4fe8ae88ae843609a66c7a2e6c7b4af P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:03.406543 1721 ts_tablet_manager.cc:1398] T e4fe8ae88ae843609a66c7a2e6c7b4af P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:03.407634 1742 tablet_bootstrap.cc:492] T ac4a753bca5f44058199942f05ef181d P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:03.407876 1742 ts_tablet_manager.cc:1367] T ac4a753bca5f44058199942f05ef181d P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.099s user 0.003s sys 0.002s
I20240503 02:13:03.409773 1742 raft_consensus.cc:348] T ac4a753bca5f44058199942f05ef181d P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:03.410080 1742 raft_consensus.cc:374] T ac4a753bca5f44058199942f05ef181d P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:03.410220 1742 raft_consensus.cc:729] T ac4a753bca5f44058199942f05ef181d P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:03.410558 1742 consensus_queue.cc:260] T ac4a753bca5f44058199942f05ef181d P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:03.411320 1742 ts_tablet_manager.cc:1398] T ac4a753bca5f44058199942f05ef181d P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 02:13:03.415372 29731 catalog_manager.cc:5484] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9094240860e43bfb129193f3bfd5be6 (127.22.108.3). New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:03.443192 1708 tablet_copy_client.cc:308] T 6bf84757d1fb44689da830a82e857787 P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.4:33935
I20240503 02:13:03.444113 30286 tablet_copy_service.cc:140] P 1f10f80f13c84e80b03eda7085a3b012: Received BeginTabletCopySession request for tablet 6bf84757d1fb44689da830a82e857787 from peer 9a4d92bc25374fb3960fde7d10be309d ({username='slave'} at 127.22.108.5:44575)
I20240503 02:13:03.444355 30286 tablet_copy_service.cc:161] P 1f10f80f13c84e80b03eda7085a3b012: Beginning new tablet copy session on tablet 6bf84757d1fb44689da830a82e857787 from peer 9a4d92bc25374fb3960fde7d10be309d at {username='slave'} at 127.22.108.5:44575: session id = 9a4d92bc25374fb3960fde7d10be309d-6bf84757d1fb44689da830a82e857787
I20240503 02:13:03.457284 30286 tablet_copy_source_session.cc:205] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:03.459148 1708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bf84757d1fb44689da830a82e857787. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.470660 1708 tablet_copy_client.cc:783] T 6bf84757d1fb44689da830a82e857787 P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:03.470924 1708 tablet_copy_client.cc:647] T 6bf84757d1fb44689da830a82e857787 P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:03.472456 1708 tablet_copy_client.cc:523] T 6bf84757d1fb44689da830a82e857787 P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:03.480221 1708 tablet_bootstrap.cc:492] T 6bf84757d1fb44689da830a82e857787 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:03.493279 1753 tablet_copy_client.cc:308] T df37d88053e2419fbe40d03023d0c756 P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.1:36719
I20240503 02:13:03.494374 29895 tablet_copy_service.cc:140] P 16440e365430405fb10366ab70e2b429: Received BeginTabletCopySession request for tablet df37d88053e2419fbe40d03023d0c756 from peer fbbc2be6382e47908ecb7c68c367bd26 ({username='slave'} at 127.22.108.7:58281)
I20240503 02:13:03.494583 29895 tablet_copy_service.cc:161] P 16440e365430405fb10366ab70e2b429: Beginning new tablet copy session on tablet df37d88053e2419fbe40d03023d0c756 from peer fbbc2be6382e47908ecb7c68c367bd26 at {username='slave'} at 127.22.108.7:58281: session id = fbbc2be6382e47908ecb7c68c367bd26-df37d88053e2419fbe40d03023d0c756
I20240503 02:13:03.498170 29895 tablet_copy_source_session.cc:205] T df37d88053e2419fbe40d03023d0c756 P 16440e365430405fb10366ab70e2b429: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:03.500268 1753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df37d88053e2419fbe40d03023d0c756. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.508071 1753 tablet_copy_client.cc:783] T df37d88053e2419fbe40d03023d0c756 P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:03.508334 1753 tablet_copy_client.cc:647] T df37d88053e2419fbe40d03023d0c756 P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:03.511312 1753 tablet_copy_client.cc:523] T df37d88053e2419fbe40d03023d0c756 P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:03.515473 1733 tablet_bootstrap.cc:492] T 779a6685fc8c4670a71b51c4493bc2f7 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:03.516009 1733 tablet_bootstrap.cc:492] T 779a6685fc8c4670a71b51c4493bc2f7 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap complete.
I20240503 02:13:03.516563 1753 tablet_bootstrap.cc:492] T df37d88053e2419fbe40d03023d0c756 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:03.521091 1733 ts_tablet_manager.cc:1367] T 779a6685fc8c4670a71b51c4493bc2f7 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.234s user 0.052s sys 0.021s
I20240503 02:13:03.529965 1733 raft_consensus.cc:348] T 779a6685fc8c4670a71b51c4493bc2f7 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:03.530315 1733 raft_consensus.cc:729] T 779a6685fc8c4670a71b51c4493bc2f7 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:03.530823 1183 raft_consensus.cc:482] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:03.531024 1183 raft_consensus.cc:504] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:03.531723 1183 leader_election.cc:288] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:13:03.532491 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4a753bca5f44058199942f05ef181d" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:13:03.532799 30264 raft_consensus.cc:2454] T ac4a753bca5f44058199942f05ef181d P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:13:03.534037 32709 leader_election.cc:302] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, 8a666e798fe8477d8dd1e51b385d5ead; no voters:
I20240503 02:13:03.534011 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4a753bca5f44058199942f05ef181d" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:03.534269 567 raft_consensus.cc:2454] T ac4a753bca5f44058199942f05ef181d P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:13:03.535001 1183 raft_consensus.cc:2781] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:03.535097 1183 raft_consensus.cc:482] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:03.535187 1183 raft_consensus.cc:3037] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:03.538169 1183 raft_consensus.cc:504] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:03.539424 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4a753bca5f44058199942f05ef181d" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:13:03.539693 30264 raft_consensus.cc:3037] T ac4a753bca5f44058199942f05ef181d P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:03.542702 30264 raft_consensus.cc:2454] T ac4a753bca5f44058199942f05ef181d P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:13:03.543504 32709 leader_election.cc:302] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, 8a666e798fe8477d8dd1e51b385d5ead; no voters:
I20240503 02:13:03.543872 1569 raft_consensus.cc:2781] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:03.546490 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4a753bca5f44058199942f05ef181d" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:13:03.546581 1183 leader_election.cc:288] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Requested vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:13:03.546777 567 raft_consensus.cc:3037] T ac4a753bca5f44058199942f05ef181d P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:03.549836 567 raft_consensus.cc:2454] T ac4a753bca5f44058199942f05ef181d P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:13:03.551107 1733 consensus_queue.cc:260] T 779a6685fc8c4670a71b51c4493bc2f7 P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:03.553705 1733 ts_tablet_manager.cc:1398] T 779a6685fc8c4670a71b51c4493bc2f7 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.032s user 0.000s sys 0.002s
I20240503 02:13:03.554445 30156 tablet_copy_service.cc:342] P a9094240860e43bfb129193f3bfd5be6: Request end of tablet copy session f575ce92c0224eeb80b9fae5f6561fbe-779a6685fc8c4670a71b51c4493bc2f7 received from {username='slave'} at 127.22.108.8:37273
I20240503 02:13:03.554602 30156 tablet_copy_service.cc:433] P a9094240860e43bfb129193f3bfd5be6: ending tablet copy session f575ce92c0224eeb80b9fae5f6561fbe-779a6685fc8c4670a71b51c4493bc2f7 on tablet 779a6685fc8c4670a71b51c4493bc2f7 with peer f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:13:03.563484 1778 ts_tablet_manager.cc:902] T 32a2dbf80daf40d6b21442a39f62f14c P 8a666e798fe8477d8dd1e51b385d5ead: Initiating tablet copy from peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:13:03.565243 1569 raft_consensus.cc:686] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Becoming Leader. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Running, Role: LEADER
I20240503 02:13:03.565690 1569 consensus_queue.cc:237] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:03.578714 29725 catalog_manager.cc:5484] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6). New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:03.587862 32640 raft_consensus.cc:1203] T 6bd4fcc4faf44d148a03beaafc1c05ba P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:03.592674 1708 tablet_bootstrap.cc:492] T 6bf84757d1fb44689da830a82e857787 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:03.593106 1708 tablet_bootstrap.cc:492] T 6bf84757d1fb44689da830a82e857787 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap complete.
I20240503 02:13:03.593361 1708 ts_tablet_manager.cc:1367] T 6bf84757d1fb44689da830a82e857787 P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.113s user 0.044s sys 0.016s
I20240503 02:13:03.594048 1708 raft_consensus.cc:348] T 6bf84757d1fb44689da830a82e857787 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:03.594246 1708 raft_consensus.cc:729] T 6bf84757d1fb44689da830a82e857787 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:03.594484 1708 consensus_queue.cc:260] T 6bf84757d1fb44689da830a82e857787 P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:03.595017 1708 ts_tablet_manager.cc:1398] T 6bf84757d1fb44689da830a82e857787 P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:13:03.595661 30286 tablet_copy_service.cc:342] P 1f10f80f13c84e80b03eda7085a3b012: Request end of tablet copy session 9a4d92bc25374fb3960fde7d10be309d-6bf84757d1fb44689da830a82e857787 received from {username='slave'} at 127.22.108.5:44575
I20240503 02:13:03.595788 30286 tablet_copy_service.cc:433] P 1f10f80f13c84e80b03eda7085a3b012: ending tablet copy session 9a4d92bc25374fb3960fde7d10be309d-6bf84757d1fb44689da830a82e857787 on tablet 6bf84757d1fb44689da830a82e857787 with peer 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:13:03.614733 1650 raft_consensus.cc:482] T 28f3f05f0a344cd683cb9e010273f908 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:03.614989 1650 raft_consensus.cc:504] T 28f3f05f0a344cd683cb9e010273f908 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:03.616043 1650 leader_election.cc:288] T 28f3f05f0a344cd683cb9e010273f908 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:13:03.616443 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28f3f05f0a344cd683cb9e010273f908" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:13:03.616729 436 raft_consensus.cc:2454] T 28f3f05f0a344cd683cb9e010273f908 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 0.
I20240503 02:13:03.617503 32574 leader_election.cc:302] T 28f3f05f0a344cd683cb9e010273f908 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:03.617839 1650 raft_consensus.cc:2781] T 28f3f05f0a344cd683cb9e010273f908 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:03.617940 1650 raft_consensus.cc:482] T 28f3f05f0a344cd683cb9e010273f908 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:03.618029 1650 raft_consensus.cc:3037] T 28f3f05f0a344cd683cb9e010273f908 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:03.619231 1778 tablet_copy_client.cc:308] T 32a2dbf80daf40d6b21442a39f62f14c P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.1:36719
I20240503 02:13:03.620014 29895 tablet_copy_service.cc:140] P 16440e365430405fb10366ab70e2b429: Received BeginTabletCopySession request for tablet 32a2dbf80daf40d6b21442a39f62f14c from peer 8a666e798fe8477d8dd1e51b385d5ead ({username='slave'} at 127.22.108.6:49099)
I20240503 02:13:03.620183 29895 tablet_copy_service.cc:161] P 16440e365430405fb10366ab70e2b429: Beginning new tablet copy session on tablet 32a2dbf80daf40d6b21442a39f62f14c from peer 8a666e798fe8477d8dd1e51b385d5ead at {username='slave'} at 127.22.108.6:49099: session id = 8a666e798fe8477d8dd1e51b385d5ead-32a2dbf80daf40d6b21442a39f62f14c
I20240503 02:13:03.620851 1650 raft_consensus.cc:504] T 28f3f05f0a344cd683cb9e010273f908 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:03.621282 30135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28f3f05f0a344cd683cb9e010273f908" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:13:03.621531 30135 raft_consensus.cc:2454] T 28f3f05f0a344cd683cb9e010273f908 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 0.
I20240503 02:13:03.623047 29895 tablet_copy_source_session.cc:205] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:03.623149 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28f3f05f0a344cd683cb9e010273f908" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:13:03.623387 436 raft_consensus.cc:3037] T 28f3f05f0a344cd683cb9e010273f908 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:03.624421 1778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32a2dbf80daf40d6b21442a39f62f14c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.626163 436 raft_consensus.cc:2454] T 28f3f05f0a344cd683cb9e010273f908 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 1.
I20240503 02:13:03.627682 32574 leader_election.cc:302] T 28f3f05f0a344cd683cb9e010273f908 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:03.628990 1650 leader_election.cc:288] T 28f3f05f0a344cd683cb9e010273f908 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 election: Requested vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:13:03.629079 30135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28f3f05f0a344cd683cb9e010273f908" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:13:03.629276 30135 raft_consensus.cc:3037] T 28f3f05f0a344cd683cb9e010273f908 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:03.629441 1650 raft_consensus.cc:2781] T 28f3f05f0a344cd683cb9e010273f908 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:03.629601 1650 raft_consensus.cc:686] T 28f3f05f0a344cd683cb9e010273f908 P 9a4d92bc25374fb3960fde7d10be309d [term 1 LEADER]: Becoming Leader. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Running, Role: LEADER
I20240503 02:13:03.629938 1650 consensus_queue.cc:237] T 28f3f05f0a344cd683cb9e010273f908 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:03.632054 30135 raft_consensus.cc:2454] T 28f3f05f0a344cd683cb9e010273f908 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 1.
I20240503 02:13:03.635025 1778 tablet_copy_client.cc:783] T 32a2dbf80daf40d6b21442a39f62f14c P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:03.635357 1778 tablet_copy_client.cc:647] T 32a2dbf80daf40d6b21442a39f62f14c P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:03.636829 1788 ts_tablet_manager.cc:902] T e61579e74a1a4a9dbd9a09f9b34d9c8d P fbbc2be6382e47908ecb7c68c367bd26: Initiating tablet copy from peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:13:03.637348 1778 tablet_copy_client.cc:523] T 32a2dbf80daf40d6b21442a39f62f14c P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:03.640772 1778 tablet_bootstrap.cc:492] T 32a2dbf80daf40d6b21442a39f62f14c P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:03.642999 1753 tablet_bootstrap.cc:492] T df37d88053e2419fbe40d03023d0c756 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:03.643515 1753 tablet_bootstrap.cc:492] T df37d88053e2419fbe40d03023d0c756 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap complete.
I20240503 02:13:03.643842 1753 ts_tablet_manager.cc:1367] T df37d88053e2419fbe40d03023d0c756 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.127s user 0.069s sys 0.004s
I20240503 02:13:03.647334 29725 catalog_manager.cc:5484] T 28f3f05f0a344cd683cb9e010273f908 P 9a4d92bc25374fb3960fde7d10be309d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5). New cstate: current_term: 1 leader_uuid: "9a4d92bc25374fb3960fde7d10be309d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:03.654136 1753 raft_consensus.cc:348] T df37d88053e2419fbe40d03023d0c756 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:03.654515 1753 raft_consensus.cc:729] T df37d88053e2419fbe40d03023d0c756 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:03.654889 1753 consensus_queue.cc:260] T df37d88053e2419fbe40d03023d0c756 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:03.657014 1753 ts_tablet_manager.cc:1398] T df37d88053e2419fbe40d03023d0c756 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20240503 02:13:03.657223 1650 raft_consensus.cc:482] T 4ab3b073d9f845cdb76a8b4aa0929924 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:03.657400 1650 raft_consensus.cc:504] T 4ab3b073d9f845cdb76a8b4aa0929924 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:03.658242 1650 leader_election.cc:288] T 4ab3b073d9f845cdb76a8b4aa0929924 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:03.658895 29895 tablet_copy_service.cc:342] P 16440e365430405fb10366ab70e2b429: Request end of tablet copy session fbbc2be6382e47908ecb7c68c367bd26-df37d88053e2419fbe40d03023d0c756 received from {username='slave'} at 127.22.108.7:58281
I20240503 02:13:03.659050 29895 tablet_copy_service.cc:433] P 16440e365430405fb10366ab70e2b429: ending tablet copy session fbbc2be6382e47908ecb7c68c367bd26-df37d88053e2419fbe40d03023d0c756 on tablet df37d88053e2419fbe40d03023d0c756 with peer fbbc2be6382e47908ecb7c68c367bd26
I20240503 02:13:03.672456 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ab3b073d9f845cdb76a8b4aa0929924" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:13:03.672771 436 raft_consensus.cc:2454] T 4ab3b073d9f845cdb76a8b4aa0929924 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 0.
I20240503 02:13:03.673954 32574 leader_election.cc:302] T 4ab3b073d9f845cdb76a8b4aa0929924 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:03.674257 1650 raft_consensus.cc:2781] T 4ab3b073d9f845cdb76a8b4aa0929924 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:03.674374 1650 raft_consensus.cc:482] T 4ab3b073d9f845cdb76a8b4aa0929924 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:03.674466 1650 raft_consensus.cc:3037] T 4ab3b073d9f845cdb76a8b4aa0929924 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:03.677309 1650 raft_consensus.cc:504] T 4ab3b073d9f845cdb76a8b4aa0929924 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:03.677994 1650 leader_election.cc:288] T 4ab3b073d9f845cdb76a8b4aa0929924 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 election: Requested vote from peers 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:03.678740 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ab3b073d9f845cdb76a8b4aa0929924" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:13:03.678975 436 raft_consensus.cc:3037] T 4ab3b073d9f845cdb76a8b4aa0929924 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:03.681751 436 raft_consensus.cc:2454] T 4ab3b073d9f845cdb76a8b4aa0929924 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 1.
I20240503 02:13:03.681815 567 raft_consensus.cc:1203] T 88b57085de3144cea3f64922ae5ccf55 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:03.683043 32574 leader_election.cc:302] T 4ab3b073d9f845cdb76a8b4aa0929924 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:03.683400 1650 raft_consensus.cc:2781] T 4ab3b073d9f845cdb76a8b4aa0929924 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:03.683557 1650 raft_consensus.cc:686] T 4ab3b073d9f845cdb76a8b4aa0929924 P 9a4d92bc25374fb3960fde7d10be309d [term 1 LEADER]: Becoming Leader. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Running, Role: LEADER
I20240503 02:13:03.683859 1650 consensus_queue.cc:237] T 4ab3b073d9f845cdb76a8b4aa0929924 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:03.691141 29725 catalog_manager.cc:5484] T 4ab3b073d9f845cdb76a8b4aa0929924 P 9a4d92bc25374fb3960fde7d10be309d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5). New cstate: current_term: 1 leader_uuid: "9a4d92bc25374fb3960fde7d10be309d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:03.692641 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ab3b073d9f845cdb76a8b4aa0929924" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:13:03.692893 305 raft_consensus.cc:2454] T 4ab3b073d9f845cdb76a8b4aa0929924 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 0.
I20240503 02:13:03.694969 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ab3b073d9f845cdb76a8b4aa0929924" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:13:03.695178 305 raft_consensus.cc:3037] T 4ab3b073d9f845cdb76a8b4aa0929924 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:03.698104 305 raft_consensus.cc:2454] T 4ab3b073d9f845cdb76a8b4aa0929924 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 1.
I20240503 02:13:03.710150 567 raft_consensus.cc:1203] T 779a6685fc8c4670a71b51c4493bc2f7 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:03.725937 1788 tablet_copy_client.cc:308] T e61579e74a1a4a9dbd9a09f9b34d9c8d P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.2:37873
I20240503 02:13:03.728099 1778 tablet_bootstrap.cc:492] T 32a2dbf80daf40d6b21442a39f62f14c P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:03.728535 1778 tablet_bootstrap.cc:492] T 32a2dbf80daf40d6b21442a39f62f14c P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap complete.
I20240503 02:13:03.728828 1778 ts_tablet_manager.cc:1367] T 32a2dbf80daf40d6b21442a39f62f14c P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.088s user 0.055s sys 0.010s
I20240503 02:13:03.729718 1778 raft_consensus.cc:348] T 32a2dbf80daf40d6b21442a39f62f14c P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:03.729985 1778 raft_consensus.cc:729] T 32a2dbf80daf40d6b21442a39f62f14c P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:03.730229 1778 consensus_queue.cc:260] T 32a2dbf80daf40d6b21442a39f62f14c P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:03.730743 1778 ts_tablet_manager.cc:1398] T 32a2dbf80daf40d6b21442a39f62f14c P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 02:13:03.731654 30026 tablet_copy_service.cc:140] P 155735c7f8b14997ac4c770aee2a9401: Received BeginTabletCopySession request for tablet e61579e74a1a4a9dbd9a09f9b34d9c8d from peer fbbc2be6382e47908ecb7c68c367bd26 ({username='slave'} at 127.22.108.7:60369)
I20240503 02:13:03.731827 30026 tablet_copy_service.cc:161] P 155735c7f8b14997ac4c770aee2a9401: Beginning new tablet copy session on tablet e61579e74a1a4a9dbd9a09f9b34d9c8d from peer fbbc2be6382e47908ecb7c68c367bd26 at {username='slave'} at 127.22.108.7:60369: session id = fbbc2be6382e47908ecb7c68c367bd26-e61579e74a1a4a9dbd9a09f9b34d9c8d
I20240503 02:13:03.732371 29895 tablet_copy_service.cc:342] P 16440e365430405fb10366ab70e2b429: Request end of tablet copy session 8a666e798fe8477d8dd1e51b385d5ead-32a2dbf80daf40d6b21442a39f62f14c received from {username='slave'} at 127.22.108.6:49099
I20240503 02:13:03.732522 29895 tablet_copy_service.cc:433] P 16440e365430405fb10366ab70e2b429: ending tablet copy session 8a666e798fe8477d8dd1e51b385d5ead-32a2dbf80daf40d6b21442a39f62f14c on tablet 32a2dbf80daf40d6b21442a39f62f14c with peer 8a666e798fe8477d8dd1e51b385d5ead
I20240503 02:13:03.734777 30026 tablet_copy_source_session.cc:205] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 155735c7f8b14997ac4c770aee2a9401: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:03.736166 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e61579e74a1a4a9dbd9a09f9b34d9c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.742365 1788 tablet_copy_client.cc:783] T e61579e74a1a4a9dbd9a09f9b34d9c8d P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:03.742647 1788 tablet_copy_client.cc:647] T e61579e74a1a4a9dbd9a09f9b34d9c8d P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:03.748898 32641 raft_consensus.cc:1203] T 6bf84757d1fb44689da830a82e857787 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:03.749787 1788 tablet_copy_client.cc:523] T e61579e74a1a4a9dbd9a09f9b34d9c8d P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:03.754424 1788 tablet_bootstrap.cc:492] T e61579e74a1a4a9dbd9a09f9b34d9c8d P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:03.777560 436 raft_consensus.cc:1203] T df37d88053e2419fbe40d03023d0c756 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:03.783684 1733 ts_tablet_manager.cc:902] T 1c9200fa969a46e6a48c64293801382b P f575ce92c0224eeb80b9fae5f6561fbe: Initiating tablet copy from peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:13:03.793386 1733 tablet_copy_client.cc:308] T 1c9200fa969a46e6a48c64293801382b P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.4:33935
I20240503 02:13:03.794034 30286 tablet_copy_service.cc:140] P 1f10f80f13c84e80b03eda7085a3b012: Received BeginTabletCopySession request for tablet 1c9200fa969a46e6a48c64293801382b from peer f575ce92c0224eeb80b9fae5f6561fbe ({username='slave'} at 127.22.108.8:51621)
I20240503 02:13:03.794217 30286 tablet_copy_service.cc:161] P 1f10f80f13c84e80b03eda7085a3b012: Beginning new tablet copy session on tablet 1c9200fa969a46e6a48c64293801382b from peer f575ce92c0224eeb80b9fae5f6561fbe at {username='slave'} at 127.22.108.8:51621: session id = f575ce92c0224eeb80b9fae5f6561fbe-1c9200fa969a46e6a48c64293801382b
I20240503 02:13:03.796980 30286 tablet_copy_source_session.cc:205] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:03.798444 1569 raft_consensus.cc:482] T e4fe8ae88ae843609a66c7a2e6c7b4af P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:03.798645 1569 raft_consensus.cc:504] T e4fe8ae88ae843609a66c7a2e6c7b4af P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:03.798799 1733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c9200fa969a46e6a48c64293801382b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.799499 1569 leader_election.cc:288] T e4fe8ae88ae843609a66c7a2e6c7b4af P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:13:03.800407 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4fe8ae88ae843609a66c7a2e6c7b4af" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:03.800694 567 raft_consensus.cc:2379] T e4fe8ae88ae843609a66c7a2e6c7b4af P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8a666e798fe8477d8dd1e51b385d5ead in current term 1: Already voted for candidate a9094240860e43bfb129193f3bfd5be6 in this term.
I20240503 02:13:03.805696 1733 tablet_copy_client.cc:783] T 1c9200fa969a46e6a48c64293801382b P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:03.805928 1733 tablet_copy_client.cc:647] T 1c9200fa969a46e6a48c64293801382b P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:03.807554 1733 tablet_copy_client.cc:523] T 1c9200fa969a46e6a48c64293801382b P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:03.822917 1788 tablet_bootstrap.cc:492] T e61579e74a1a4a9dbd9a09f9b34d9c8d P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:03.823400 1788 tablet_bootstrap.cc:492] T e61579e74a1a4a9dbd9a09f9b34d9c8d P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap complete.
I20240503 02:13:03.824146 1733 tablet_bootstrap.cc:492] T 1c9200fa969a46e6a48c64293801382b P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:03.824421 1788 ts_tablet_manager.cc:1367] T e61579e74a1a4a9dbd9a09f9b34d9c8d P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.070s user 0.048s sys 0.009s
I20240503 02:13:03.825492 1788 raft_consensus.cc:348] T e61579e74a1a4a9dbd9a09f9b34d9c8d P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:03.825821 1788 raft_consensus.cc:729] T e61579e74a1a4a9dbd9a09f9b34d9c8d P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:03.826110 1788 consensus_queue.cc:260] T e61579e74a1a4a9dbd9a09f9b34d9c8d P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:03.827230 1788 ts_tablet_manager.cc:1398] T e61579e74a1a4a9dbd9a09f9b34d9c8d P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:03.828022 30026 tablet_copy_service.cc:342] P 155735c7f8b14997ac4c770aee2a9401: Request end of tablet copy session fbbc2be6382e47908ecb7c68c367bd26-e61579e74a1a4a9dbd9a09f9b34d9c8d received from {username='slave'} at 127.22.108.7:60369
I20240503 02:13:03.828186 30026 tablet_copy_service.cc:433] P 155735c7f8b14997ac4c770aee2a9401: ending tablet copy session fbbc2be6382e47908ecb7c68c367bd26-e61579e74a1a4a9dbd9a09f9b34d9c8d on tablet e61579e74a1a4a9dbd9a09f9b34d9c8d with peer fbbc2be6382e47908ecb7c68c367bd26
I20240503 02:13:03.856447 30135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4fe8ae88ae843609a66c7a2e6c7b4af" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:13:03.857362 32708 leader_election.cc:302] T e4fe8ae88ae843609a66c7a2e6c7b4af P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead; no voters: a9094240860e43bfb129193f3bfd5be6, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:13:03.857707 1569 raft_consensus.cc:3037] T e4fe8ae88ae843609a66c7a2e6c7b4af P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:03.860438 1569 raft_consensus.cc:2726] T e4fe8ae88ae843609a66c7a2e6c7b4af P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:03.863337 29725 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:13:03.866865 1760 consensus_queue.cc:1035] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:03.917966 436 raft_consensus.cc:1261] T 4ab3b073d9f845cdb76a8b4aa0929924 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:03.918140 305 raft_consensus.cc:1261] T 4ab3b073d9f845cdb76a8b4aa0929924 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:03.918850 1650 consensus_queue.cc:1035] T 4ab3b073d9f845cdb76a8b4aa0929924 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:03.919359 1650 consensus_queue.cc:1035] T 4ab3b073d9f845cdb76a8b4aa0929924 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:03.947575 29731 catalog_manager.cc:5576] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975) does not have the latest schema for tablet 4ab3b073d9f845cdb76a8b4aa0929924 (table rebalancer_dynamic_table [id=99a9a7da96cb474a8c0a1a83c7fbdb4b]). Expected version 1 got 0
I20240503 02:13:03.949132 1831 consensus_queue.cc:1035] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:03.974377 1837 tablet.cc:1722] T 4ab3b073d9f845cdb76a8b4aa0929924 P 9a4d92bc25374fb3960fde7d10be309d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:03.974996 1837 tablet.cc:1620] T 4ab3b073d9f845cdb76a8b4aa0929924 P 9a4d92bc25374fb3960fde7d10be309d: MemRowSet was empty: no flush needed.
I20240503 02:13:03.975697 1838 tablet.cc:1722] T 4ab3b073d9f845cdb76a8b4aa0929924 P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:03.976270 1838 tablet.cc:1620] T 4ab3b073d9f845cdb76a8b4aa0929924 P 8a666e798fe8477d8dd1e51b385d5ead: MemRowSet was empty: no flush needed.
I20240503 02:13:03.982357 29731 catalog_manager.cc:5576] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241) does not have the latest schema for tablet 4ab3b073d9f845cdb76a8b4aa0929924 (table rebalancer_dynamic_table [id=99a9a7da96cb474a8c0a1a83c7fbdb4b]). Expected version 1 got 0
I20240503 02:13:03.985101 29725 catalog_manager.cc:5576] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975) does not have the latest schema for tablet 4ab3b073d9f845cdb76a8b4aa0929924 (table rebalancer_dynamic_table [id=99a9a7da96cb474a8c0a1a83c7fbdb4b]). Expected version 1 got 0
I20240503 02:13:03.987368 1733 tablet_bootstrap.cc:492] T 1c9200fa969a46e6a48c64293801382b P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:03.987865 1733 tablet_bootstrap.cc:492] T 1c9200fa969a46e6a48c64293801382b P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap complete.
I20240503 02:13:03.988209 1733 ts_tablet_manager.cc:1367] T 1c9200fa969a46e6a48c64293801382b P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.175s user 0.048s sys 0.008s
I20240503 02:13:03.990067 1733 raft_consensus.cc:348] T 1c9200fa969a46e6a48c64293801382b P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:03.990324 1733 raft_consensus.cc:729] T 1c9200fa969a46e6a48c64293801382b P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:03.990571 1733 consensus_queue.cc:260] T 1c9200fa969a46e6a48c64293801382b P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:03.991310 1733 ts_tablet_manager.cc:1398] T 1c9200fa969a46e6a48c64293801382b P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:03.993842 30286 tablet_copy_service.cc:342] P 1f10f80f13c84e80b03eda7085a3b012: Request end of tablet copy session f575ce92c0224eeb80b9fae5f6561fbe-1c9200fa969a46e6a48c64293801382b received from {username='slave'} at 127.22.108.8:51621
I20240503 02:13:03.993973 30286 tablet_copy_service.cc:433] P 1f10f80f13c84e80b03eda7085a3b012: ending tablet copy session f575ce92c0224eeb80b9fae5f6561fbe-1c9200fa969a46e6a48c64293801382b on tablet 1c9200fa969a46e6a48c64293801382b with peer f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:13:04.001595 29730 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=99a9a7da96cb474a8c0a1a83c7fbdb4b] alter complete (version 1)
I20240503 02:13:04.007252 1841 tablet.cc:1722] T 4ab3b073d9f845cdb76a8b4aa0929924 P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:04.007798 1841 tablet.cc:1620] T 4ab3b073d9f845cdb76a8b4aa0929924 P fbbc2be6382e47908ecb7c68c367bd26: MemRowSet was empty: no flush needed.
I20240503 02:13:04.010401 29725 catalog_manager.cc:5576] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241) does not have the latest schema for tablet 4ab3b073d9f845cdb76a8b4aa0929924 (table rebalancer_dynamic_table [id=99a9a7da96cb474a8c0a1a83c7fbdb4b]). Expected version 1 got 0
I20240503 02:13:04.033877 1787 consensus_queue.cc:1035] T 28f3f05f0a344cd683cb9e010273f908 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:04.054417 1650 consensus_queue.cc:1035] T 28f3f05f0a344cd683cb9e010273f908 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:04.098188 29853 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:13:04.108414 29986 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:04.113526 30245 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:04.122407 29725 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:13:04.122874 1569 consensus_queue.cc:1035] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:04.128898 32755 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:04.146354 305 raft_consensus.cc:1203] T 32a2dbf80daf40d6b21442a39f62f14c P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:04.148392 1837 tablet.cc:1722] T 4ab3b073d9f845cdb76a8b4aa0929924 P 9a4d92bc25374fb3960fde7d10be309d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:04.148988 1837 tablet.cc:1620] T 4ab3b073d9f845cdb76a8b4aa0929924 P 9a4d92bc25374fb3960fde7d10be309d: MemRowSet was empty: no flush needed.
I20240503 02:13:04.154985 1841 tablet.cc:1722] T 4ab3b073d9f845cdb76a8b4aa0929924 P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:04.155586 1841 tablet.cc:1620] T 4ab3b073d9f845cdb76a8b4aa0929924 P fbbc2be6382e47908ecb7c68c367bd26: MemRowSet was empty: no flush needed.
I20240503 02:13:04.163758 1569 consensus_queue.cc:1035] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:04.164384 1838 tablet.cc:1722] T 4ab3b073d9f845cdb76a8b4aa0929924 P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:04.164945 1838 tablet.cc:1620] T 4ab3b073d9f845cdb76a8b4aa0929924 P 8a666e798fe8477d8dd1e51b385d5ead: MemRowSet was empty: no flush needed.
I20240503 02:13:04.189672 29731 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=99a9a7da96cb474a8c0a1a83c7fbdb4b] alter complete (version 2)
I20240503 02:13:04.222885 29731 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_extra_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 02:13:04.238482 29940 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.7:44241, user_credentials={real_user=slave}} blocked reactor thread for 104982us
I20240503 02:13:04.238845 568 raft_consensus.cc:1203] T 1c9200fa969a46e6a48c64293801382b P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:04.253762 32620 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:13:04.255002 29986 tablet_service.cc:1467] Processing CreateTablet for tablet dafe0f997f23405ca0a1bc6c0f7eab42 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=97e0c0e7261b408f9cb42ad58ba75222]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:04.255157 32755 tablet_service.cc:1467] Processing CreateTablet for tablet dafe0f997f23405ca0a1bc6c0f7eab42 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=97e0c0e7261b408f9cb42ad58ba75222]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:04.255483 29986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dafe0f997f23405ca0a1bc6c0f7eab42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:04.255738 32755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dafe0f997f23405ca0a1bc6c0f7eab42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:04.255786 547 tablet_service.cc:1467] Processing CreateTablet for tablet dafe0f997f23405ca0a1bc6c0f7eab42 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=97e0c0e7261b408f9cb42ad58ba75222]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:04.256193 547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dafe0f997f23405ca0a1bc6c0f7eab42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:04.257366 546 tablet_service.cc:1467] Processing CreateTablet for tablet 62d5e9ea7af8435bb34028df2b3f1457 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=97e0c0e7261b408f9cb42ad58ba75222]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:04.257782 546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62d5e9ea7af8435bb34028df2b3f1457. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:04.257716 417 tablet_service.cc:1467] Processing CreateTablet for tablet 62d5e9ea7af8435bb34028df2b3f1457 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=97e0c0e7261b408f9cb42ad58ba75222]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:04.257901 32620 tablet_service.cc:1467] Processing CreateTablet for tablet 62d5e9ea7af8435bb34028df2b3f1457 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=97e0c0e7261b408f9cb42ad58ba75222]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:04.258178 417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62d5e9ea7af8435bb34028df2b3f1457. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:04.258316 32620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62d5e9ea7af8435bb34028df2b3f1457. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:04.264207 30116 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:04.283233 32619 tablet_service.cc:1467] Processing CreateTablet for tablet 9fa3f1e452c24c7d83a36f10284b8b5d (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=97e0c0e7261b408f9cb42ad58ba75222]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:04.283885 32619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fa3f1e452c24c7d83a36f10284b8b5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:04.291816 1872 tablet_bootstrap.cc:492] T dafe0f997f23405ca0a1bc6c0f7eab42 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:13:04.292193 29733 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:13:04.300100 1871 tablet_bootstrap.cc:492] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:04.302145 1872 tablet_bootstrap.cc:654] T dafe0f997f23405ca0a1bc6c0f7eab42 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:04.307420 415 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:13:04.307476 30245 tablet_service.cc:1467] Processing CreateTablet for tablet 9fa3f1e452c24c7d83a36f10284b8b5d (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=97e0c0e7261b408f9cb42ad58ba75222]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:04.307955 30245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fa3f1e452c24c7d83a36f10284b8b5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:04.309109 1871 tablet_bootstrap.cc:654] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:04.320356 1871 tablet_bootstrap.cc:492] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:04.320582 417 tablet_service.cc:1467] Processing CreateTablet for tablet 9fa3f1e452c24c7d83a36f10284b8b5d (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=97e0c0e7261b408f9cb42ad58ba75222]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:04.321532 417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fa3f1e452c24c7d83a36f10284b8b5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:04.328519 1871 ts_tablet_manager.cc:1367] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.003s
I20240503 02:13:04.331001 1871 raft_consensus.cc:348] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:04.331696 1871 raft_consensus.cc:374] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:04.331965 1871 raft_consensus.cc:729] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:04.332499 1871 consensus_queue.cc:260] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:04.333642 1871 ts_tablet_manager.cc:1398] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:13:04.341145 1873 tablet_bootstrap.cc:492] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:04.343338 1873 tablet_bootstrap.cc:654] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:04.345443 1880 tablet_bootstrap.cc:492] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:13:04.347656 1880 tablet_bootstrap.cc:654] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:04.349124 1874 tablet_bootstrap.cc:492] T 62d5e9ea7af8435bb34028df2b3f1457 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:04.351235 1874 tablet_bootstrap.cc:654] T 62d5e9ea7af8435bb34028df2b3f1457 P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:04.363058 1871 tablet_bootstrap.cc:492] T 9fa3f1e452c24c7d83a36f10284b8b5d P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:04.364918 1871 tablet_bootstrap.cc:654] T 9fa3f1e452c24c7d83a36f10284b8b5d P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:04.371698 548 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:13:04.386469 1880 tablet_bootstrap.cc:492] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:13:04.386746 1880 ts_tablet_manager.cc:1367] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.000s
I20240503 02:13:04.387629 1880 raft_consensus.cc:348] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:04.387929 1880 raft_consensus.cc:374] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:04.388065 1880 raft_consensus.cc:729] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:13:04.388726 1880 consensus_queue.cc:260] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:04.390394 1880 ts_tablet_manager.cc:1398] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:04.390897 1837 tablet.cc:1722] T 4ab3b073d9f845cdb76a8b4aa0929924 P 9a4d92bc25374fb3960fde7d10be309d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240503 02:13:04.400676 29939 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.3:42679, user_credentials={real_user=slave}} blocked reactor thread for 197949us
I20240503 02:13:04.407867 436 raft_consensus.cc:1203] T e61579e74a1a4a9dbd9a09f9b34d9c8d P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240503 02:13:04.408543 32574 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.7:44241, user_credentials={real_user=slave}} blocked reactor thread for 94407.6us
I20240503 02:13:04.411013 1838 tablet.cc:1722] T 4ab3b073d9f845cdb76a8b4aa0929924 P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:04.414958 1873 tablet_bootstrap.cc:492] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:04.415127 1873 ts_tablet_manager.cc:1367] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.074s user 0.007s sys 0.007s
I20240503 02:13:04.415134 1874 tablet_bootstrap.cc:492] T 62d5e9ea7af8435bb34028df2b3f1457 P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:04.415271 1874 ts_tablet_manager.cc:1367] T 62d5e9ea7af8435bb34028df2b3f1457 P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.066s user 0.006s sys 0.007s
I20240503 02:13:04.415946 1873 raft_consensus.cc:348] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:04.415973 1874 raft_consensus.cc:348] T 62d5e9ea7af8435bb34028df2b3f1457 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:04.416152 1873 raft_consensus.cc:374] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:04.416249 1873 raft_consensus.cc:729] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:04.416553 1873 consensus_queue.cc:260] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:04.417171 1873 ts_tablet_manager.cc:1398] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:13:04.418913 1841 tablet.cc:1722] T 4ab3b073d9f845cdb76a8b4aa0929924 P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:04.423175 1872 tablet_bootstrap.cc:492] T dafe0f997f23405ca0a1bc6c0f7eab42 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:13:04.423372 1872 ts_tablet_manager.cc:1367] T dafe0f997f23405ca0a1bc6c0f7eab42 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.132s user 0.006s sys 0.000s
I20240503 02:13:04.429602 1874 raft_consensus.cc:374] T 62d5e9ea7af8435bb34028df2b3f1457 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:04.429849 1874 raft_consensus.cc:729] T 62d5e9ea7af8435bb34028df2b3f1457 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:04.429767 1872 raft_consensus.cc:348] T dafe0f997f23405ca0a1bc6c0f7eab42 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:04.429994 1872 raft_consensus.cc:374] T dafe0f997f23405ca0a1bc6c0f7eab42 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:04.429986 29728 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table_new_name [id=99a9a7da96cb474a8c0a1a83c7fbdb4b] alter complete (version 3)
I20240503 02:13:04.430090 1872 raft_consensus.cc:729] T dafe0f997f23405ca0a1bc6c0f7eab42 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:13:04.433707 1872 consensus_queue.cc:260] T dafe0f997f23405ca0a1bc6c0f7eab42 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:04.434687 1872 ts_tablet_manager.cc:1398] T dafe0f997f23405ca0a1bc6c0f7eab42 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20240503 02:13:04.445514 1874 consensus_queue.cc:260] T 62d5e9ea7af8435bb34028df2b3f1457 P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:04.449223 1874 ts_tablet_manager.cc:1398] T 62d5e9ea7af8435bb34028df2b3f1457 P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.034s user 0.001s sys 0.001s
I20240503 02:13:04.450351 1871 tablet_bootstrap.cc:492] T 9fa3f1e452c24c7d83a36f10284b8b5d P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:04.450601 1871 ts_tablet_manager.cc:1367] T 9fa3f1e452c24c7d83a36f10284b8b5d P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.088s user 0.009s sys 0.004s
I20240503 02:13:04.451543 1871 raft_consensus.cc:348] T 9fa3f1e452c24c7d83a36f10284b8b5d P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:04.451937 1871 raft_consensus.cc:374] T 9fa3f1e452c24c7d83a36f10284b8b5d P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:04.452068 1871 raft_consensus.cc:729] T 9fa3f1e452c24c7d83a36f10284b8b5d P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:04.452365 1871 consensus_queue.cc:260] T 9fa3f1e452c24c7d83a36f10284b8b5d P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:04.453251 1871 ts_tablet_manager.cc:1398] T 9fa3f1e452c24c7d83a36f10284b8b5d P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:13:04.453649 1874 tablet_bootstrap.cc:492] T 9fa3f1e452c24c7d83a36f10284b8b5d P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:04.455502 1874 tablet_bootstrap.cc:654] T 9fa3f1e452c24c7d83a36f10284b8b5d P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:04.460397 1887 tablet_bootstrap.cc:492] T dafe0f997f23405ca0a1bc6c0f7eab42 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:04.462258 1887 tablet_bootstrap.cc:654] T dafe0f997f23405ca0a1bc6c0f7eab42 P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:04.474891 1864 raft_consensus.cc:482] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:04.475096 1864 raft_consensus.cc:504] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:04.476305 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dafe0f997f23405ca0a1bc6c0f7eab42" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:04.476596 1864 leader_election.cc:288] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:13:04.477142 30006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dafe0f997f23405ca0a1bc6c0f7eab42" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:13:04.477437 30006 raft_consensus.cc:2454] T dafe0f997f23405ca0a1bc6c0f7eab42 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
W20240503 02:13:04.478093 32708 leader_election.cc:341] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:04.478603 32709 leader_election.cc:302] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 8a666e798fe8477d8dd1e51b385d5ead; no voters: f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:13:04.481143 1864 raft_consensus.cc:2781] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:04.481492 1864 raft_consensus.cc:482] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:04.481653 1864 raft_consensus.cc:3037] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:04.484921 1864 raft_consensus.cc:504] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:04.487632 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dafe0f997f23405ca0a1bc6c0f7eab42" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
W20240503 02:13:04.488385 32708 leader_election.cc:341] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:04.489287 30006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dafe0f997f23405ca0a1bc6c0f7eab42" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:13:04.489600 30006 raft_consensus.cc:3037] T dafe0f997f23405ca0a1bc6c0f7eab42 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:04.493196 30006 raft_consensus.cc:2454] T dafe0f997f23405ca0a1bc6c0f7eab42 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:13:04.493655 32709 leader_election.cc:302] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 8a666e798fe8477d8dd1e51b385d5ead; no voters: f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:13:04.493973 1885 raft_consensus.cc:2781] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:04.494124 1885 raft_consensus.cc:686] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Becoming Leader. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Running, Role: LEADER
I20240503 02:13:04.494441 1885 consensus_queue.cc:237] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:04.504678 1874 tablet_bootstrap.cc:492] T 9fa3f1e452c24c7d83a36f10284b8b5d P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:04.505131 1874 ts_tablet_manager.cc:1367] T 9fa3f1e452c24c7d83a36f10284b8b5d P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.052s user 0.010s sys 0.017s
I20240503 02:13:04.506424 1874 raft_consensus.cc:348] T 9fa3f1e452c24c7d83a36f10284b8b5d P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:04.507828 1874 raft_consensus.cc:374] T 9fa3f1e452c24c7d83a36f10284b8b5d P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:04.508101 1874 raft_consensus.cc:729] T 9fa3f1e452c24c7d83a36f10284b8b5d P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:04.508471 1874 consensus_queue.cc:260] T 9fa3f1e452c24c7d83a36f10284b8b5d P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:04.509894 1864 leader_election.cc:288] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Requested vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:13:04.510622 1874 ts_tablet_manager.cc:1398] T 9fa3f1e452c24c7d83a36f10284b8b5d P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:13:04.521579 29728 catalog_manager.cc:5484] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6). New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:04.522274 1887 tablet_bootstrap.cc:492] T dafe0f997f23405ca0a1bc6c0f7eab42 P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:04.522439 1887 ts_tablet_manager.cc:1367] T dafe0f997f23405ca0a1bc6c0f7eab42 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.062s user 0.004s sys 0.000s
I20240503 02:13:04.523434 1887 raft_consensus.cc:348] T dafe0f997f23405ca0a1bc6c0f7eab42 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:04.523710 1887 raft_consensus.cc:374] T dafe0f997f23405ca0a1bc6c0f7eab42 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:04.523844 1887 raft_consensus.cc:729] T dafe0f997f23405ca0a1bc6c0f7eab42 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:04.524118 1887 consensus_queue.cc:260] T dafe0f997f23405ca0a1bc6c0f7eab42 P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:04.524703 1887 ts_tablet_manager.cc:1398] T dafe0f997f23405ca0a1bc6c0f7eab42 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:04.531025 1887 tablet_bootstrap.cc:492] T 62d5e9ea7af8435bb34028df2b3f1457 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:04.532918 1887 tablet_bootstrap.cc:654] T 62d5e9ea7af8435bb34028df2b3f1457 P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:04.538549 29730 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:04.539044 29730 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:04.547107 29730 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet 4ab3b073d9f845cdb76a8b4aa0929924
I20240503 02:13:04.548728 32755 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ab3b073d9f845cdb76a8b4aa0929924 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:04 UTC) from {username='slave'} at 127.0.0.1:46360
I20240503 02:13:04.549168 417 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ab3b073d9f845cdb76a8b4aa0929924 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:04 UTC) from {username='slave'} at 127.0.0.1:50814
I20240503 02:13:04.549551 32619 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ab3b073d9f845cdb76a8b4aa0929924 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:04 UTC) from {username='slave'} at 127.0.0.1:38200
I20240503 02:13:04.553965 1887 tablet_bootstrap.cc:492] T 62d5e9ea7af8435bb34028df2b3f1457 P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:04.554160 1887 ts_tablet_manager.cc:1367] T 62d5e9ea7af8435bb34028df2b3f1457 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.000s
I20240503 02:13:04.555120 1887 raft_consensus.cc:348] T 62d5e9ea7af8435bb34028df2b3f1457 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:04.555379 1887 raft_consensus.cc:374] T 62d5e9ea7af8435bb34028df2b3f1457 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:04.555445 1887 raft_consensus.cc:729] T 62d5e9ea7af8435bb34028df2b3f1457 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:04.555696 1887 consensus_queue.cc:260] T 62d5e9ea7af8435bb34028df2b3f1457 P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:04.556291 1887 ts_tablet_manager.cc:1398] T 62d5e9ea7af8435bb34028df2b3f1457 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:13:04.560575 1893 tablet_replica.cc:330] T 4ab3b073d9f845cdb76a8b4aa0929924 P fbbc2be6382e47908ecb7c68c367bd26: stopping tablet replica
I20240503 02:13:04.561218 1892 tablet_replica.cc:330] T 4ab3b073d9f845cdb76a8b4aa0929924 P 8a666e798fe8477d8dd1e51b385d5ead: stopping tablet replica
I20240503 02:13:04.562111 1894 tablet_replica.cc:330] T 4ab3b073d9f845cdb76a8b4aa0929924 P 9a4d92bc25374fb3960fde7d10be309d: stopping tablet replica
I20240503 02:13:04.562412 1893 raft_consensus.cc:2229] T 4ab3b073d9f845cdb76a8b4aa0929924 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:04.562747 1892 raft_consensus.cc:2229] T 4ab3b073d9f845cdb76a8b4aa0929924 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:04.564023 1892 raft_consensus.cc:2258] T 4ab3b073d9f845cdb76a8b4aa0929924 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:04.563547 1893 raft_consensus.cc:2258] T 4ab3b073d9f845cdb76a8b4aa0929924 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:04.564334 1894 raft_consensus.cc:2229] T 4ab3b073d9f845cdb76a8b4aa0929924 P 9a4d92bc25374fb3960fde7d10be309d [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:04.565639 1894 raft_consensus.cc:2258] T 4ab3b073d9f845cdb76a8b4aa0929924 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:04.566538 1893 ts_tablet_manager.cc:1875] T 4ab3b073d9f845cdb76a8b4aa0929924 P fbbc2be6382e47908ecb7c68c367bd26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:04.567390 1892 ts_tablet_manager.cc:1875] T 4ab3b073d9f845cdb76a8b4aa0929924 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:04.574941 1806 raft_consensus.cc:482] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:04.575104 1806 raft_consensus.cc:504] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:04.575907 1806 leader_election.cc:288] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:13:04.577085 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d5e9ea7af8435bb34028df2b3f1457" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:04.577324 567 raft_consensus.cc:2454] T 62d5e9ea7af8435bb34028df2b3f1457 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 0.
I20240503 02:13:04.577786 1533 raft_consensus.cc:482] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:04.577960 1533 raft_consensus.cc:504] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:04.578469 370 leader_election.cc:302] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f575ce92c0224eeb80b9fae5f6561fbe, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:04.578569 1533 leader_election.cc:288] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:04.579593 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d5e9ea7af8435bb34028df2b3f1457" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:13:04.579881 32640 raft_consensus.cc:2454] T 62d5e9ea7af8435bb34028df2b3f1457 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 0.
I20240503 02:13:04.580340 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fa3f1e452c24c7d83a36f10284b8b5d" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:13:04.580586 436 raft_consensus.cc:2454] T 9fa3f1e452c24c7d83a36f10284b8b5d P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:13:04.581771 30200 leader_election.cc:302] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:04.581880 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fa3f1e452c24c7d83a36f10284b8b5d" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:13:04.582113 1533 raft_consensus.cc:2781] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:04.582103 32640 raft_consensus.cc:2454] T 9fa3f1e452c24c7d83a36f10284b8b5d P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:13:04.582195 1533 raft_consensus.cc:482] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:04.582285 1533 raft_consensus.cc:3037] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:04.582893 1806 raft_consensus.cc:2781] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:04.583016 1806 raft_consensus.cc:482] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:04.583109 1806 raft_consensus.cc:3037] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:04.585803 1533 raft_consensus.cc:504] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:04.587023 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fa3f1e452c24c7d83a36f10284b8b5d" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:13:04.587218 436 raft_consensus.cc:3037] T 9fa3f1e452c24c7d83a36f10284b8b5d P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:04.588328 1533 leader_election.cc:288] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Requested vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:04.588748 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fa3f1e452c24c7d83a36f10284b8b5d" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d"
I20240503 02:13:04.588989 32640 raft_consensus.cc:3037] T 9fa3f1e452c24c7d83a36f10284b8b5d P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:04.590090 436 raft_consensus.cc:2454] T 9fa3f1e452c24c7d83a36f10284b8b5d P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:13:04.590579 30200 leader_election.cc:302] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:04.590951 1533 raft_consensus.cc:2781] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:04.591547 1533 raft_consensus.cc:686] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Becoming Leader. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Running, Role: LEADER
I20240503 02:13:04.596475 1806 raft_consensus.cc:504] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:04.596976 32640 raft_consensus.cc:2454] T 9fa3f1e452c24c7d83a36f10284b8b5d P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:13:04.598662 1806 leader_election.cc:288] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Requested vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:13:04.604387 1894 ts_tablet_manager.cc:1875] T 4ab3b073d9f845cdb76a8b4aa0929924 P 9a4d92bc25374fb3960fde7d10be309d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:04.592042 1533 consensus_queue.cc:237] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:04.605515 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d5e9ea7af8435bb34028df2b3f1457" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d"
I20240503 02:13:04.613387 32640 raft_consensus.cc:3037] T 62d5e9ea7af8435bb34028df2b3f1457 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:04.616993 32640 raft_consensus.cc:2454] T 62d5e9ea7af8435bb34028df2b3f1457 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 1.
I20240503 02:13:04.619279 371 leader_election.cc:302] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:04.619832 1806 raft_consensus.cc:2781] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:04.621752 1806 raft_consensus.cc:686] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Becoming Leader. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Running, Role: LEADER
I20240503 02:13:04.623275 1806 consensus_queue.cc:237] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:04.653924 29730 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:13:04.663683 29728 catalog_manager.cc:5484] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4). New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:04.664608 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d5e9ea7af8435bb34028df2b3f1457" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:13:04.664902 567 raft_consensus.cc:3037] T 62d5e9ea7af8435bb34028df2b3f1457 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:04.668597 567 raft_consensus.cc:2454] T 62d5e9ea7af8435bb34028df2b3f1457 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 1.
I20240503 02:13:04.680613 32619 tablet_service.cc:1467] Processing CreateTablet for tablet e4d8164736d1409a933b06391c27bd8d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=11c3a184b65d48b6b02c35ba6d559c83]), partition=
I20240503 02:13:04.681152 32619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4d8164736d1409a933b06391c27bd8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:04.681933 29853 tablet_service.cc:1467] Processing CreateTablet for tablet e4d8164736d1409a933b06391c27bd8d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=11c3a184b65d48b6b02c35ba6d559c83]), partition=
I20240503 02:13:04.682310 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4d8164736d1409a933b06391c27bd8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:04.684741 547 tablet_service.cc:1467] Processing CreateTablet for tablet e4d8164736d1409a933b06391c27bd8d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=11c3a184b65d48b6b02c35ba6d559c83]), partition=
I20240503 02:13:04.685098 547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4d8164736d1409a933b06391c27bd8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:04.689460 29730 catalog_manager.cc:5484] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7). New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:04.697321 1902 tablet_bootstrap.cc:492] T e4d8164736d1409a933b06391c27bd8d P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:13:04.697871 1887 tablet_bootstrap.cc:492] T e4d8164736d1409a933b06391c27bd8d P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:04.699476 1887 tablet_bootstrap.cc:654] T e4d8164736d1409a933b06391c27bd8d P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:04.700894 1861 raft_consensus.cc:482] T dafe0f997f23405ca0a1bc6c0f7eab42 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:04.700978 1902 tablet_bootstrap.cc:654] T e4d8164736d1409a933b06391c27bd8d P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:04.701079 1861 raft_consensus.cc:504] T dafe0f997f23405ca0a1bc6c0f7eab42 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:04.702229 1861 leader_election.cc:288] T dafe0f997f23405ca0a1bc6c0f7eab42 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:04.702764 1874 tablet_bootstrap.cc:492] T e4d8164736d1409a933b06391c27bd8d P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:04.704623 1874 tablet_bootstrap.cc:654] T e4d8164736d1409a933b06391c27bd8d P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:04.713956 1887 tablet_bootstrap.cc:492] T e4d8164736d1409a933b06391c27bd8d P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:04.714298 1887 ts_tablet_manager.cc:1367] T e4d8164736d1409a933b06391c27bd8d P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.000s
I20240503 02:13:04.715250 1887 raft_consensus.cc:348] T e4d8164736d1409a933b06391c27bd8d P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:04.715554 1887 raft_consensus.cc:374] T e4d8164736d1409a933b06391c27bd8d P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:04.716099 1887 raft_consensus.cc:729] T e4d8164736d1409a933b06391c27bd8d P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:04.716437 1887 consensus_queue.cc:260] T e4d8164736d1409a933b06391c27bd8d P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:04.717111 1887 ts_tablet_manager.cc:1398] T e4d8164736d1409a933b06391c27bd8d P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:04.742668 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dafe0f997f23405ca0a1bc6c0f7eab42" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:13:04.747555 1894 ts_tablet_manager.cc:1888] T 4ab3b073d9f845cdb76a8b4aa0929924 P 9a4d92bc25374fb3960fde7d10be309d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:04.767928 1892 ts_tablet_manager.cc:1888] T 4ab3b073d9f845cdb76a8b4aa0929924 P 8a666e798fe8477d8dd1e51b385d5ead: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:04.768234 1892 log.cc:1196] T 4ab3b073d9f845cdb76a8b4aa0929924 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal/wals/4ab3b073d9f845cdb76a8b4aa0929924
I20240503 02:13:04.768895 1892 ts_tablet_manager.cc:1909] T 4ab3b073d9f845cdb76a8b4aa0929924 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting consensus metadata
I20240503 02:13:04.770577 1893 ts_tablet_manager.cc:1888] T 4ab3b073d9f845cdb76a8b4aa0929924 P fbbc2be6382e47908ecb7c68c367bd26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:04.770725 29719 catalog_manager.cc:4830] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): tablet 4ab3b073d9f845cdb76a8b4aa0929924 (table rebalancer_dynamic_table_new_name [id=99a9a7da96cb474a8c0a1a83c7fbdb4b]) successfully deleted
I20240503 02:13:04.771040 1893 log.cc:1196] T 4ab3b073d9f845cdb76a8b4aa0929924 P fbbc2be6382e47908ecb7c68c367bd26: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal/wals/4ab3b073d9f845cdb76a8b4aa0929924
I20240503 02:13:04.772119 1893 ts_tablet_manager.cc:1909] T 4ab3b073d9f845cdb76a8b4aa0929924 P fbbc2be6382e47908ecb7c68c367bd26: Deleting consensus metadata
I20240503 02:13:04.750370 1874 tablet_bootstrap.cc:492] T e4d8164736d1409a933b06391c27bd8d P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:04.772382 1902 tablet_bootstrap.cc:492] T e4d8164736d1409a933b06391c27bd8d P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:13:04.772423 1874 ts_tablet_manager.cc:1367] T e4d8164736d1409a933b06391c27bd8d P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.070s user 0.009s sys 0.003s
I20240503 02:13:04.772624 1902 ts_tablet_manager.cc:1367] T e4d8164736d1409a933b06391c27bd8d P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.075s user 0.021s sys 0.013s
I20240503 02:13:04.773375 1874 raft_consensus.cc:348] T e4d8164736d1409a933b06391c27bd8d P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:04.773624 1874 raft_consensus.cc:374] T e4d8164736d1409a933b06391c27bd8d P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:04.773423 1902 raft_consensus.cc:348] T e4d8164736d1409a933b06391c27bd8d P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:04.773718 1874 raft_consensus.cc:729] T e4d8164736d1409a933b06391c27bd8d P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:04.773810 1902 raft_consensus.cc:374] T e4d8164736d1409a933b06391c27bd8d P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:04.773962 1902 raft_consensus.cc:729] T e4d8164736d1409a933b06391c27bd8d P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:13:04.773967 1874 consensus_queue.cc:260] T e4d8164736d1409a933b06391c27bd8d P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:04.774647 1874 ts_tablet_manager.cc:1398] T e4d8164736d1409a933b06391c27bd8d P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:04.774791 1894 log.cc:1196] T 4ab3b073d9f845cdb76a8b4aa0929924 P 9a4d92bc25374fb3960fde7d10be309d: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal/wals/4ab3b073d9f845cdb76a8b4aa0929924
I20240503 02:13:04.774582 1902 consensus_queue.cc:260] T e4d8164736d1409a933b06391c27bd8d P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:04.775269 1894 ts_tablet_manager.cc:1909] T 4ab3b073d9f845cdb76a8b4aa0929924 P 9a4d92bc25374fb3960fde7d10be309d: Deleting consensus metadata
I20240503 02:13:04.775578 1902 ts_tablet_manager.cc:1398] T e4d8164736d1409a933b06391c27bd8d P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:04.776803 29719 catalog_manager.cc:4830] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): tablet 4ab3b073d9f845cdb76a8b4aa0929924 (table rebalancer_dynamic_table_new_name [id=99a9a7da96cb474a8c0a1a83c7fbdb4b]) successfully deleted
I20240503 02:13:04.794950 29719 catalog_manager.cc:4830] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): tablet 4ab3b073d9f845cdb76a8b4aa0929924 (table rebalancer_dynamic_table_new_name [id=99a9a7da96cb474a8c0a1a83c7fbdb4b]) successfully deleted
I20240503 02:13:04.795348 30006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dafe0f997f23405ca0a1bc6c0f7eab42" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:13:04.795694 30006 raft_consensus.cc:2379] T dafe0f997f23405ca0a1bc6c0f7eab42 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f575ce92c0224eeb80b9fae5f6561fbe in current term 1: Already voted for candidate 8a666e798fe8477d8dd1e51b385d5ead in this term.
I20240503 02:13:04.796252 502 leader_election.cc:302] T dafe0f997f23405ca0a1bc6c0f7eab42 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f575ce92c0224eeb80b9fae5f6561fbe; no voters: 155735c7f8b14997ac4c770aee2a9401, 8a666e798fe8477d8dd1e51b385d5ead
I20240503 02:13:04.797017 1861 raft_consensus.cc:3037] T dafe0f997f23405ca0a1bc6c0f7eab42 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:04.800575 1861 raft_consensus.cc:2726] T dafe0f997f23405ca0a1bc6c0f7eab42 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:04.984405 30265 consensus_queue.cc:237] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:04.986974 30134 raft_consensus.cc:1261] T 1c9200fa969a46e6a48c64293801382b P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:04.986989 567 raft_consensus.cc:1261] T 1c9200fa969a46e6a48c64293801382b P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:04.988273 1896 consensus_queue.cc:1035] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:04.988577 1533 consensus_queue.cc:1035] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:04.991153 1533 raft_consensus.cc:2932] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:04.991756 566 raft_consensus.cc:2932] T 1c9200fa969a46e6a48c64293801382b P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:04.997329 29730 catalog_manager.cc:5484] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: config changed from index 4 to 5, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:05.003818 1885 consensus_queue.cc:1035] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:05.004290 29853 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c9200fa969a46e6a48c64293801382b with delete_type TABLET_DATA_TOMBSTONED (TS 16440e365430405fb10366ab70e2b429 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:58744
I20240503 02:13:05.008523 1927 tablet_replica.cc:330] T 1c9200fa969a46e6a48c64293801382b P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:13:05.010846 1927 raft_consensus.cc:2229] T 1c9200fa969a46e6a48c64293801382b P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:05.011090 1927 raft_consensus.cc:2258] T 1c9200fa969a46e6a48c64293801382b P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:05.011708 1927 ts_tablet_manager.cc:1875] T 1c9200fa969a46e6a48c64293801382b P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:05.023299 1934 consensus_queue.cc:1035] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:05.023347 30134 raft_consensus.cc:2932] T 1c9200fa969a46e6a48c64293801382b P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:05.030421 1533 consensus_queue.cc:1035] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:05.036656 1927 ts_tablet_manager.cc:1888] T 1c9200fa969a46e6a48c64293801382b P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:13:05.036943 1927 log.cc:1196] T 1c9200fa969a46e6a48c64293801382b P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/1c9200fa969a46e6a48c64293801382b
I20240503 02:13:05.038085 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet 1c9200fa969a46e6a48c64293801382b (table rebalance_test_table_15 [id=6bbf46811ff140d289e600493149f118]) successfully deleted
I20240503 02:13:05.049156 1533 consensus_queue.cc:1035] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:05.063099 1806 consensus_queue.cc:1035] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:05.071765 1895 consensus_queue.cc:1035] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:05.140661 1861 raft_consensus.cc:482] T e4d8164736d1409a933b06391c27bd8d P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:05.140880 1861 raft_consensus.cc:504] T e4d8164736d1409a933b06391c27bd8d P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:05.141559 1861 leader_election.cc:288] T e4d8164736d1409a933b06391c27bd8d P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:13:05.141963 29872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4d8164736d1409a933b06391c27bd8d" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:13:05.142220 29872 raft_consensus.cc:2454] T e4d8164736d1409a933b06391c27bd8d P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 0.
I20240503 02:13:05.142932 504 leader_election.cc:302] T e4d8164736d1409a933b06391c27bd8d P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:05.143117 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4d8164736d1409a933b06391c27bd8d" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:13:05.143405 32640 raft_consensus.cc:2454] T e4d8164736d1409a933b06391c27bd8d P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 0.
I20240503 02:13:05.143510 1909 raft_consensus.cc:2781] T e4d8164736d1409a933b06391c27bd8d P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:05.143601 1909 raft_consensus.cc:482] T e4d8164736d1409a933b06391c27bd8d P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:05.143687 1909 raft_consensus.cc:3037] T e4d8164736d1409a933b06391c27bd8d P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:05.146551 1909 raft_consensus.cc:504] T e4d8164736d1409a933b06391c27bd8d P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:05.147922 29872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4d8164736d1409a933b06391c27bd8d" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:13:05.148135 29872 raft_consensus.cc:3037] T e4d8164736d1409a933b06391c27bd8d P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:05.149731 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4d8164736d1409a933b06391c27bd8d" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d"
I20240503 02:13:05.149971 32640 raft_consensus.cc:3037] T e4d8164736d1409a933b06391c27bd8d P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:05.152247 32640 raft_consensus.cc:2454] T e4d8164736d1409a933b06391c27bd8d P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 1.
I20240503 02:13:05.152679 502 leader_election.cc:302] T e4d8164736d1409a933b06391c27bd8d P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:05.153061 1861 raft_consensus.cc:2781] T e4d8164736d1409a933b06391c27bd8d P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:05.154713 29872 raft_consensus.cc:2454] T e4d8164736d1409a933b06391c27bd8d P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 1.
I20240503 02:13:05.155076 1909 leader_election.cc:288] T e4d8164736d1409a933b06391c27bd8d P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 election: Requested vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:13:05.155323 1861 raft_consensus.cc:686] T e4d8164736d1409a933b06391c27bd8d P f575ce92c0224eeb80b9fae5f6561fbe [term 1 LEADER]: Becoming Leader. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Running, Role: LEADER
I20240503 02:13:05.155591 1861 consensus_queue.cc:237] T e4d8164736d1409a933b06391c27bd8d P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:05.159083 29728 catalog_manager.cc:5484] T e4d8164736d1409a933b06391c27bd8d P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: term changed from 0 to 1, leader changed from <none> to f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8). New cstate: current_term: 1 leader_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:05.252341 29986 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:05.252628 29732 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_extra_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:05.266464 32619 tablet_service.cc:1467] Processing CreateTablet for tablet c42a89fd9daf427bb5a60b4ee76a9287 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=3e0093624a2d4e6fa34244d26a961331]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:05.266980 32619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c42a89fd9daf427bb5a60b4ee76a9287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:05.267215 29853 tablet_service.cc:1467] Processing CreateTablet for tablet c97c7a843366426887115e13f35faf52 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=3e0093624a2d4e6fa34244d26a961331]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:05.267642 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c97c7a843366426887115e13f35faf52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:05.269500 32620 tablet_service.cc:1467] Processing CreateTablet for tablet c97c7a843366426887115e13f35faf52 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=3e0093624a2d4e6fa34244d26a961331]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:05.269999 32620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c97c7a843366426887115e13f35faf52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:05.270125 547 tablet_service.cc:1467] Processing CreateTablet for tablet c42a89fd9daf427bb5a60b4ee76a9287 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=3e0093624a2d4e6fa34244d26a961331]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:05.270586 547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c42a89fd9daf427bb5a60b4ee76a9287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:05.272876 30245 tablet_service.cc:1467] Processing CreateTablet for tablet 2a132e4747824d5989858e649bc243c2 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=3e0093624a2d4e6fa34244d26a961331]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:05.273330 30245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a132e4747824d5989858e649bc243c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:05.273350 30116 tablet_service.cc:1467] Processing CreateTablet for tablet 2a132e4747824d5989858e649bc243c2 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=3e0093624a2d4e6fa34244d26a961331]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:05.273763 30116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a132e4747824d5989858e649bc243c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:05.274463 32621 tablet_service.cc:1467] Processing CreateTablet for tablet 2a132e4747824d5989858e649bc243c2 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=3e0093624a2d4e6fa34244d26a961331]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:05.274830 32621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a132e4747824d5989858e649bc243c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:05.275251 1902 tablet_bootstrap.cc:492] T c97c7a843366426887115e13f35faf52 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:13:05.276890 1902 tablet_bootstrap.cc:654] T c97c7a843366426887115e13f35faf52 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:05.277451 32755 tablet_service.cc:1467] Processing CreateTablet for tablet c97c7a843366426887115e13f35faf52 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=3e0093624a2d4e6fa34244d26a961331]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:05.277580 29986 tablet_service.cc:1467] Processing CreateTablet for tablet c42a89fd9daf427bb5a60b4ee76a9287 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=3e0093624a2d4e6fa34244d26a961331]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:05.277889 32755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c97c7a843366426887115e13f35faf52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:05.278030 29986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c42a89fd9daf427bb5a60b4ee76a9287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:05.284612 29853 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:13:05.295420 29732 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:13:05.306006 30244 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:05.309722 32640 raft_consensus.cc:1261] T e4d8164736d1409a933b06391c27bd8d P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:05.310345 1909 consensus_queue.cc:1035] T e4d8164736d1409a933b06391c27bd8d P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:05.312165 29872 raft_consensus.cc:1261] T e4d8164736d1409a933b06391c27bd8d P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:05.312848 1909 consensus_queue.cc:1035] T e4d8164736d1409a933b06391c27bd8d P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:05.317994 1992 tablet_bootstrap.cc:492] T c97c7a843366426887115e13f35faf52 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:05.321568 1990 tablet_bootstrap.cc:492] T c42a89fd9daf427bb5a60b4ee76a9287 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:05.326349 1992 tablet_bootstrap.cc:654] T c97c7a843366426887115e13f35faf52 P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:05.328482 1993 tablet_bootstrap.cc:492] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:13:05.332832 1990 tablet_bootstrap.cc:654] T c42a89fd9daf427bb5a60b4ee76a9287 P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:05.339308 1990 tablet_bootstrap.cc:492] T c42a89fd9daf427bb5a60b4ee76a9287 P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:05.341017 1990 ts_tablet_manager.cc:1367] T c42a89fd9daf427bb5a60b4ee76a9287 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.000s
I20240503 02:13:05.342634 1990 raft_consensus.cc:348] T c42a89fd9daf427bb5a60b4ee76a9287 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:05.343349 1993 tablet_bootstrap.cc:654] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:05.349565 1991 tablet_bootstrap.cc:492] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:13:05.351816 1991 tablet_bootstrap.cc:654] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:05.352407 1989 tablet_bootstrap.cc:492] T c97c7a843366426887115e13f35faf52 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:05.355489 1989 tablet_bootstrap.cc:654] T c97c7a843366426887115e13f35faf52 P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:05.361380 1990 raft_consensus.cc:374] T c42a89fd9daf427bb5a60b4ee76a9287 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:05.361742 1992 tablet_bootstrap.cc:492] T c97c7a843366426887115e13f35faf52 P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:05.361918 1990 raft_consensus.cc:729] T c42a89fd9daf427bb5a60b4ee76a9287 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:05.362224 1992 ts_tablet_manager.cc:1367] T c97c7a843366426887115e13f35faf52 P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.044s user 0.006s sys 0.001s
I20240503 02:13:05.362614 1990 consensus_queue.cc:260] T c42a89fd9daf427bb5a60b4ee76a9287 P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:05.363912 1992 raft_consensus.cc:348] T c97c7a843366426887115e13f35faf52 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:05.366837 1994 tablet_bootstrap.cc:492] T 2a132e4747824d5989858e649bc243c2 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:13:05.370834 1990 ts_tablet_manager.cc:1398] T c42a89fd9daf427bb5a60b4ee76a9287 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20240503 02:13:05.371233 29732 catalog_manager.cc:5576] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595) does not have the latest schema for tablet e4d8164736d1409a933b06391c27bd8d (table rebalancer_dynamic_table [id=11c3a184b65d48b6b02c35ba6d559c83]). Expected version 1 got 0
I20240503 02:13:05.371387 1992 raft_consensus.cc:374] T c97c7a843366426887115e13f35faf52 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:05.371786 1992 raft_consensus.cc:729] T c97c7a843366426887115e13f35faf52 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:05.372256 1992 consensus_queue.cc:260] T c97c7a843366426887115e13f35faf52 P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:05.373513 1992 ts_tablet_manager.cc:1398] T c97c7a843366426887115e13f35faf52 P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240503 02:13:05.375744 1992 tablet_bootstrap.cc:492] T 2a132e4747824d5989858e649bc243c2 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:05.378249 1902 tablet_bootstrap.cc:492] T c97c7a843366426887115e13f35faf52 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:13:05.381562 29731 catalog_manager.cc:5576] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719) does not have the latest schema for tablet e4d8164736d1409a933b06391c27bd8d (table rebalancer_dynamic_table [id=11c3a184b65d48b6b02c35ba6d559c83]). Expected version 1 got 0
I20240503 02:13:05.382519 1992 tablet_bootstrap.cc:654] T 2a132e4747824d5989858e649bc243c2 P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:05.382819 1902 ts_tablet_manager.cc:1367] T c97c7a843366426887115e13f35faf52 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.108s user 0.016s sys 0.022s
I20240503 02:13:05.384821 1902 raft_consensus.cc:348] T c97c7a843366426887115e13f35faf52 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:05.385196 1902 raft_consensus.cc:374] T c97c7a843366426887115e13f35faf52 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:05.385324 1902 raft_consensus.cc:729] T c97c7a843366426887115e13f35faf52 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:13:05.385134 2002 tablet.cc:1722] T e4d8164736d1409a933b06391c27bd8d P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:05.385654 1902 consensus_queue.cc:260] T c97c7a843366426887115e13f35faf52 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:05.386720 1902 ts_tablet_manager.cc:1398] T c97c7a843366426887115e13f35faf52 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:13:05.390583 2002 tablet.cc:1620] T e4d8164736d1409a933b06391c27bd8d P f575ce92c0224eeb80b9fae5f6561fbe: MemRowSet was empty: no flush needed.
I20240503 02:13:05.407374 2007 tablet.cc:1722] T e4d8164736d1409a933b06391c27bd8d P 16440e365430405fb10366ab70e2b429: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:05.412434 29725 catalog_manager.cc:5576] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719) does not have the latest schema for tablet e4d8164736d1409a933b06391c27bd8d (table rebalancer_dynamic_table [id=11c3a184b65d48b6b02c35ba6d559c83]). Expected version 1 got 0
I20240503 02:13:05.414561 2003 tablet.cc:1722] T e4d8164736d1409a933b06391c27bd8d P 9a4d92bc25374fb3960fde7d10be309d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:05.415221 2007 tablet.cc:1620] T e4d8164736d1409a933b06391c27bd8d P 16440e365430405fb10366ab70e2b429: MemRowSet was empty: no flush needed.
I20240503 02:13:05.417057 1484 raft_consensus.cc:482] T c97c7a843366426887115e13f35faf52 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:05.417250 1484 raft_consensus.cc:504] T c97c7a843366426887115e13f35faf52 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:05.419422 29725 catalog_manager.cc:5576] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595) does not have the latest schema for tablet e4d8164736d1409a933b06391c27bd8d (table rebalancer_dynamic_table [id=11c3a184b65d48b6b02c35ba6d559c83]). Expected version 1 got 0
I20240503 02:13:05.432677 29731 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=11c3a184b65d48b6b02c35ba6d559c83] alter complete (version 1)
I20240503 02:13:05.438221 32755 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:13:05.439759 2003 tablet.cc:1620] T e4d8164736d1409a933b06391c27bd8d P 9a4d92bc25374fb3960fde7d10be309d: MemRowSet was empty: no flush needed.
I20240503 02:13:05.440042 1484 leader_election.cc:288] T c97c7a843366426887115e13f35faf52 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:13:05.440732 1994 tablet_bootstrap.cc:654] T 2a132e4747824d5989858e649bc243c2 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:05.451822 1992 tablet_bootstrap.cc:492] T 2a132e4747824d5989858e649bc243c2 P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:05.454675 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97c7a843366426887115e13f35faf52" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:13:05.456580 1994 tablet_bootstrap.cc:492] T 2a132e4747824d5989858e649bc243c2 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:13:05.458261 1992 ts_tablet_manager.cc:1367] T 2a132e4747824d5989858e649bc243c2 P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.083s user 0.000s sys 0.013s
I20240503 02:13:05.458355 1994 ts_tablet_manager.cc:1367] T 2a132e4747824d5989858e649bc243c2 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.092s user 0.005s sys 0.000s
I20240503 02:13:05.459221 1994 raft_consensus.cc:348] T 2a132e4747824d5989858e649bc243c2 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:05.459512 1992 raft_consensus.cc:348] T 2a132e4747824d5989858e649bc243c2 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:05.459878 1992 raft_consensus.cc:374] T 2a132e4747824d5989858e649bc243c2 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:05.460211 1992 raft_consensus.cc:729] T 2a132e4747824d5989858e649bc243c2 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:05.460619 1992 consensus_queue.cc:260] T 2a132e4747824d5989858e649bc243c2 P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:05.461792 1992 ts_tablet_manager.cc:1398] T 2a132e4747824d5989858e649bc243c2 P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:13:05.462679 1992 tablet_bootstrap.cc:492] T c42a89fd9daf427bb5a60b4ee76a9287 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:05.466467 1994 raft_consensus.cc:374] T 2a132e4747824d5989858e649bc243c2 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:05.466604 1994 raft_consensus.cc:729] T 2a132e4747824d5989858e649bc243c2 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:13:05.466845 1994 consensus_queue.cc:260] T 2a132e4747824d5989858e649bc243c2 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:05.468216 1992 tablet_bootstrap.cc:654] T c42a89fd9daf427bb5a60b4ee76a9287 P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:05.469897 1991 tablet_bootstrap.cc:492] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:13:05.470041 1991 ts_tablet_manager.cc:1367] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.121s user 0.004s sys 0.000s
I20240503 02:13:05.470793 1991 raft_consensus.cc:348] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:05.470985 1991 raft_consensus.cc:374] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:05.471061 1991 raft_consensus.cc:729] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:13:05.471282 1991 consensus_queue.cc:260] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:05.471836 1991 ts_tablet_manager.cc:1398] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:13:05.472124 1994 ts_tablet_manager.cc:1398] T 2a132e4747824d5989858e649bc243c2 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240503 02:13:05.498188 1992 tablet_bootstrap.cc:492] T c42a89fd9daf427bb5a60b4ee76a9287 P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:05.499154 1992 ts_tablet_manager.cc:1367] T c42a89fd9daf427bb5a60b4ee76a9287 P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.004s
I20240503 02:13:05.500313 1992 raft_consensus.cc:348] T c42a89fd9daf427bb5a60b4ee76a9287 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:05.500712 1992 raft_consensus.cc:374] T c42a89fd9daf427bb5a60b4ee76a9287 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:05.500875 1992 raft_consensus.cc:729] T c42a89fd9daf427bb5a60b4ee76a9287 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:05.501191 1992 consensus_queue.cc:260] T c42a89fd9daf427bb5a60b4ee76a9287 P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:05.501355 32620 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:13:05.501789 1993 tablet_bootstrap.cc:492] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:13:05.501873 1992 ts_tablet_manager.cc:1398] T c42a89fd9daf427bb5a60b4ee76a9287 P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:05.502064 1993 ts_tablet_manager.cc:1367] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.174s user 0.012s sys 0.013s
I20240503 02:13:05.506224 1989 tablet_bootstrap.cc:492] T c97c7a843366426887115e13f35faf52 P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:05.506402 1989 ts_tablet_manager.cc:1367] T c97c7a843366426887115e13f35faf52 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.154s user 0.004s sys 0.000s
I20240503 02:13:05.506448 1976 raft_consensus.cc:482] T c42a89fd9daf427bb5a60b4ee76a9287 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:05.506574 1976 raft_consensus.cc:504] T c42a89fd9daf427bb5a60b4ee76a9287 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:05.507300 1989 raft_consensus.cc:348] T c97c7a843366426887115e13f35faf52 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:05.507409 1976 leader_election.cc:288] T c42a89fd9daf427bb5a60b4ee76a9287 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:13:05.507519 1989 raft_consensus.cc:374] T c97c7a843366426887115e13f35faf52 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:05.507627 1989 raft_consensus.cc:729] T c97c7a843366426887115e13f35faf52 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:05.507877 1989 consensus_queue.cc:260] T c97c7a843366426887115e13f35faf52 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:05.508222 566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c42a89fd9daf427bb5a60b4ee76a9287" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:05.508440 1989 ts_tablet_manager.cc:1398] T c97c7a843366426887115e13f35faf52 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:05.508456 566 raft_consensus.cc:2454] T c42a89fd9daf427bb5a60b4ee76a9287 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 0.
I20240503 02:13:05.508944 32573 leader_election.cc:302] T c42a89fd9daf427bb5a60b4ee76a9287 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:05.509322 1976 raft_consensus.cc:2781] T c42a89fd9daf427bb5a60b4ee76a9287 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:05.509428 1976 raft_consensus.cc:482] T c42a89fd9daf427bb5a60b4ee76a9287 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:05.509506 1976 raft_consensus.cc:3037] T c42a89fd9daf427bb5a60b4ee76a9287 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:05.512882 1976 raft_consensus.cc:504] T c42a89fd9daf427bb5a60b4ee76a9287 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:05.514751 566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c42a89fd9daf427bb5a60b4ee76a9287" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:13:05.514999 566 raft_consensus.cc:3037] T c42a89fd9daf427bb5a60b4ee76a9287 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:05.517904 566 raft_consensus.cc:2454] T c42a89fd9daf427bb5a60b4ee76a9287 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 1.
I20240503 02:13:05.518393 32573 leader_election.cc:302] T c42a89fd9daf427bb5a60b4ee76a9287 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:05.518684 1787 raft_consensus.cc:2781] T c42a89fd9daf427bb5a60b4ee76a9287 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:05.520833 1976 leader_election.cc:288] T c42a89fd9daf427bb5a60b4ee76a9287 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 election: Requested vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:13:05.503067 1993 raft_consensus.cc:348] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:05.539592 1993 raft_consensus.cc:374] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:05.539738 1993 raft_consensus.cc:729] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:13:05.540004 1993 consensus_queue.cc:260] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:05.540647 1993 ts_tablet_manager.cc:1398] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.038s user 0.000s sys 0.001s
I20240503 02:13:05.542364 30116 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:05.550109 29725 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:13:05.557418 417 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:13:05.575956 29809 leader_election.cc:341] T c97c7a843366426887115e13f35faf52 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:05.579706 2002 tablet.cc:1722] T e4d8164736d1409a933b06391c27bd8d P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:05.580255 2002 tablet.cc:1620] T e4d8164736d1409a933b06391c27bd8d P f575ce92c0224eeb80b9fae5f6561fbe: MemRowSet was empty: no flush needed.
I20240503 02:13:05.586460 2003 tablet.cc:1722] T e4d8164736d1409a933b06391c27bd8d P 9a4d92bc25374fb3960fde7d10be309d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:05.586989 2003 tablet.cc:1620] T e4d8164736d1409a933b06391c27bd8d P 9a4d92bc25374fb3960fde7d10be309d: MemRowSet was empty: no flush needed.
I20240503 02:13:05.606987 1787 raft_consensus.cc:686] T c42a89fd9daf427bb5a60b4ee76a9287 P 9a4d92bc25374fb3960fde7d10be309d [term 1 LEADER]: Becoming Leader. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Running, Role: LEADER
I20240503 02:13:05.607369 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97c7a843366426887115e13f35faf52" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:13:05.607640 32640 raft_consensus.cc:2454] T c97c7a843366426887115e13f35faf52 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:13:05.607611 1787 consensus_queue.cc:237] T c42a89fd9daf427bb5a60b4ee76a9287 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:05.610399 29809 leader_election.cc:302] T c97c7a843366426887115e13f35faf52 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16440e365430405fb10366ab70e2b429, 9a4d92bc25374fb3960fde7d10be309d; no voters: 8a666e798fe8477d8dd1e51b385d5ead
I20240503 02:13:05.611124 2019 raft_consensus.cc:2781] T c97c7a843366426887115e13f35faf52 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:05.611233 2019 raft_consensus.cc:482] T c97c7a843366426887115e13f35faf52 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:05.611341 2019 raft_consensus.cc:3037] T c97c7a843366426887115e13f35faf52 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:05.612100 2007 tablet.cc:1722] T e4d8164736d1409a933b06391c27bd8d P 16440e365430405fb10366ab70e2b429: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:05.612685 2007 tablet.cc:1620] T e4d8164736d1409a933b06391c27bd8d P 16440e365430405fb10366ab70e2b429: MemRowSet was empty: no flush needed.
I20240503 02:13:05.615844 29725 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=11c3a184b65d48b6b02c35ba6d559c83] alter complete (version 2)
I20240503 02:13:05.616609 2019 raft_consensus.cc:504] T c97c7a843366426887115e13f35faf52 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:05.617715 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97c7a843366426887115e13f35faf52" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:13:05.617973 307 raft_consensus.cc:3037] T c97c7a843366426887115e13f35faf52 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:05.622833 1976 raft_consensus.cc:482] T 2a132e4747824d5989858e649bc243c2 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:05.623004 1976 raft_consensus.cc:504] T 2a132e4747824d5989858e649bc243c2 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:05.623265 307 raft_consensus.cc:2454] T c97c7a843366426887115e13f35faf52 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:13:05.623764 1976 leader_election.cc:288] T 2a132e4747824d5989858e649bc243c2 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:13:05.624789 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a132e4747824d5989858e649bc243c2" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:13:05.625064 30265 raft_consensus.cc:2454] T 2a132e4747824d5989858e649bc243c2 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 0.
I20240503 02:13:05.625325 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c42a89fd9daf427bb5a60b4ee76a9287" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:13:05.625852 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a132e4747824d5989858e649bc243c2" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:13:05.625625 30005 raft_consensus.cc:3037] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:05.626099 30134 raft_consensus.cc:2454] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 0.
I20240503 02:13:05.626569 29809 leader_election.cc:302] T c97c7a843366426887115e13f35faf52 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, 8a666e798fe8477d8dd1e51b385d5ead; no voters:
I20240503 02:13:05.626919 1484 raft_consensus.cc:2781] T c97c7a843366426887115e13f35faf52 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:05.625432 30006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c42a89fd9daf427bb5a60b4ee76a9287" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:13:05.627852 32573 leader_election.cc:302] T 2a132e4747824d5989858e649bc243c2 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:13:05.630192 30005 raft_consensus.cc:2454] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 1.
I20240503 02:13:05.631541 1976 raft_consensus.cc:2781] T 2a132e4747824d5989858e649bc243c2 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:05.631664 1976 raft_consensus.cc:482] T 2a132e4747824d5989858e649bc243c2 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:05.631747 1976 raft_consensus.cc:3037] T 2a132e4747824d5989858e649bc243c2 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:05.634878 1976 raft_consensus.cc:504] T 2a132e4747824d5989858e649bc243c2 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:05.636179 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a132e4747824d5989858e649bc243c2" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:13:05.636394 30265 raft_consensus.cc:3037] T 2a132e4747824d5989858e649bc243c2 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:05.639156 30265 raft_consensus.cc:2454] T 2a132e4747824d5989858e649bc243c2 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 1.
I20240503 02:13:05.640179 32574 leader_election.cc:302] T 2a132e4747824d5989858e649bc243c2 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, 9a4d92bc25374fb3960fde7d10be309d; no voters:
I20240503 02:13:05.640537 2018 raft_consensus.cc:2781] T 2a132e4747824d5989858e649bc243c2 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:05.643308 1976 leader_election.cc:288] T 2a132e4747824d5989858e649bc243c2 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 election: Requested vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:13:05.644282 547 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:13:05.647948 29732 catalog_manager.cc:5484] T c42a89fd9daf427bb5a60b4ee76a9287 P 9a4d92bc25374fb3960fde7d10be309d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5). New cstate: current_term: 1 leader_uuid: "9a4d92bc25374fb3960fde7d10be309d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:05.650158 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a132e4747824d5989858e649bc243c2" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:13:05.650436 30134 raft_consensus.cc:3037] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:05.669987 2018 raft_consensus.cc:686] T 2a132e4747824d5989858e649bc243c2 P 9a4d92bc25374fb3960fde7d10be309d [term 1 LEADER]: Becoming Leader. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Running, Role: LEADER
I20240503 02:13:05.670420 30134 raft_consensus.cc:2454] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 1.
I20240503 02:13:05.670466 2018 consensus_queue.cc:237] T 2a132e4747824d5989858e649bc243c2 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:05.674124 1484 raft_consensus.cc:686] T c97c7a843366426887115e13f35faf52 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Becoming Leader. State: Replica: 16440e365430405fb10366ab70e2b429, State: Running, Role: LEADER
I20240503 02:13:05.674432 1484 consensus_queue.cc:237] T c97c7a843366426887115e13f35faf52 P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:05.679176 2019 leader_election.cc:288] T c97c7a843366426887115e13f35faf52 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Requested vote from peers 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:13:05.680546 29732 catalog_manager.cc:5484] T 2a132e4747824d5989858e649bc243c2 P 9a4d92bc25374fb3960fde7d10be309d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5). New cstate: current_term: 1 leader_uuid: "9a4d92bc25374fb3960fde7d10be309d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:05.681334 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97c7a843366426887115e13f35faf52" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d"
I20240503 02:13:05.681576 32641 raft_consensus.cc:3037] T c97c7a843366426887115e13f35faf52 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:05.684444 32641 raft_consensus.cc:2454] T c97c7a843366426887115e13f35faf52 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:13:05.690304 29725 catalog_manager.cc:5484] T c97c7a843366426887115e13f35faf52 P 16440e365430405fb10366ab70e2b429 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16440e365430405fb10366ab70e2b429 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:05.719390 29725 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:13:05.730115 2002 tablet.cc:1722] T e4d8164736d1409a933b06391c27bd8d P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:05.730868 2003 tablet.cc:1722] T e4d8164736d1409a933b06391c27bd8d P 9a4d92bc25374fb3960fde7d10be309d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:05.735891 2007 tablet.cc:1722] T e4d8164736d1409a933b06391c27bd8d P 16440e365430405fb10366ab70e2b429: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:05.746289 29732 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table_new_name [id=11c3a184b65d48b6b02c35ba6d559c83] alter complete (version 3)
I20240503 02:13:05.770992 30005 consensus_queue.cc:237] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:05.772931 437 raft_consensus.cc:1261] T e61579e74a1a4a9dbd9a09f9b34d9c8d P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:05.773463 29874 raft_consensus.cc:1261] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:05.773478 1865 consensus_queue.cc:1035] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:05.774026 2020 consensus_queue.cc:1035] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:05.776170 1865 raft_consensus.cc:2932] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:05.777313 29874 raft_consensus.cc:2932] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:05.782135 29731 catalog_manager.cc:5484] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:05.782454 437 raft_consensus.cc:2932] T e61579e74a1a4a9dbd9a09f9b34d9c8d P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:05.786240 30116 tablet_service.cc:1514] Processing DeleteTablet for tablet e61579e74a1a4a9dbd9a09f9b34d9c8d with delete_type TABLET_DATA_TOMBSTONED (TS a9094240860e43bfb129193f3bfd5be6 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44812
I20240503 02:13:05.788123 2049 tablet_replica.cc:330] T e61579e74a1a4a9dbd9a09f9b34d9c8d P a9094240860e43bfb129193f3bfd5be6: stopping tablet replica
I20240503 02:13:05.788347 2049 raft_consensus.cc:2229] T e61579e74a1a4a9dbd9a09f9b34d9c8d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:05.788895 2049 raft_consensus.cc:2258] T e61579e74a1a4a9dbd9a09f9b34d9c8d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:05.789716 2049 ts_tablet_manager.cc:1875] T e61579e74a1a4a9dbd9a09f9b34d9c8d P a9094240860e43bfb129193f3bfd5be6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:05.795320 2049 ts_tablet_manager.cc:1888] T e61579e74a1a4a9dbd9a09f9b34d9c8d P a9094240860e43bfb129193f3bfd5be6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:13:05.795424 2049 log.cc:1196] T e61579e74a1a4a9dbd9a09f9b34d9c8d P a9094240860e43bfb129193f3bfd5be6: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/wals/e61579e74a1a4a9dbd9a09f9b34d9c8d
I20240503 02:13:05.796190 29718 catalog_manager.cc:4830] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): tablet e61579e74a1a4a9dbd9a09f9b34d9c8d (table rebalance_test_table_13 [id=df07c1fd4e654fea9a59ea9a0ea79595]) successfully deleted
I20240503 02:13:05.847966 29731 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:05.848209 29731 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:05.855979 29731 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet e4d8164736d1409a933b06391c27bd8d
I20240503 02:13:05.856968 547 tablet_service.cc:1514] Processing DeleteTablet for tablet e4d8164736d1409a933b06391c27bd8d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:05 UTC) from {username='slave'} at 127.0.0.1:39028
I20240503 02:13:05.857700 32620 tablet_service.cc:1514] Processing DeleteTablet for tablet e4d8164736d1409a933b06391c27bd8d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:05 UTC) from {username='slave'} at 127.0.0.1:38200
I20240503 02:13:05.860569 29853 tablet_service.cc:1514] Processing DeleteTablet for tablet e4d8164736d1409a933b06391c27bd8d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:05 UTC) from {username='slave'} at 127.0.0.1:58744
I20240503 02:13:05.861464 2062 tablet_replica.cc:330] T e4d8164736d1409a933b06391c27bd8d P f575ce92c0224eeb80b9fae5f6561fbe: stopping tablet replica
I20240503 02:13:05.861817 2062 raft_consensus.cc:2229] T e4d8164736d1409a933b06391c27bd8d P f575ce92c0224eeb80b9fae5f6561fbe [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:05.862957 2062 raft_consensus.cc:2258] T e4d8164736d1409a933b06391c27bd8d P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:05.870074 2062 ts_tablet_manager.cc:1875] T e4d8164736d1409a933b06391c27bd8d P f575ce92c0224eeb80b9fae5f6561fbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:05.883719 2064 tablet_replica.cc:330] T e4d8164736d1409a933b06391c27bd8d P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:13:05.884130 2064 raft_consensus.cc:2229] T e4d8164736d1409a933b06391c27bd8d P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:05.884454 2064 raft_consensus.cc:2258] T e4d8164736d1409a933b06391c27bd8d P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:05.885502 2064 ts_tablet_manager.cc:1875] T e4d8164736d1409a933b06391c27bd8d P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:05.886135 2062 ts_tablet_manager.cc:1888] T e4d8164736d1409a933b06391c27bd8d P f575ce92c0224eeb80b9fae5f6561fbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:05.886341 2062 log.cc:1196] T e4d8164736d1409a933b06391c27bd8d P f575ce92c0224eeb80b9fae5f6561fbe: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal/wals/e4d8164736d1409a933b06391c27bd8d
I20240503 02:13:05.886881 2062 ts_tablet_manager.cc:1909] T e4d8164736d1409a933b06391c27bd8d P f575ce92c0224eeb80b9fae5f6561fbe: Deleting consensus metadata
I20240503 02:13:05.888113 29718 catalog_manager.cc:4830] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): tablet e4d8164736d1409a933b06391c27bd8d (table rebalancer_dynamic_table_new_name [id=11c3a184b65d48b6b02c35ba6d559c83]) successfully deleted
I20240503 02:13:05.893242 2064 ts_tablet_manager.cc:1888] T e4d8164736d1409a933b06391c27bd8d P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:05.893379 2064 log.cc:1196] T e4d8164736d1409a933b06391c27bd8d P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/e4d8164736d1409a933b06391c27bd8d
I20240503 02:13:05.893813 2064 ts_tablet_manager.cc:1909] T e4d8164736d1409a933b06391c27bd8d P 16440e365430405fb10366ab70e2b429: Deleting consensus metadata
I20240503 02:13:05.894673 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet e4d8164736d1409a933b06391c27bd8d (table rebalancer_dynamic_table_new_name [id=11c3a184b65d48b6b02c35ba6d559c83]) successfully deleted
I20240503 02:13:05.906818 2063 tablet_replica.cc:330] T e4d8164736d1409a933b06391c27bd8d P 9a4d92bc25374fb3960fde7d10be309d: stopping tablet replica
I20240503 02:13:05.907335 2063 raft_consensus.cc:2229] T e4d8164736d1409a933b06391c27bd8d P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:05.907582 2063 raft_consensus.cc:2258] T e4d8164736d1409a933b06391c27bd8d P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:05.908558 2063 ts_tablet_manager.cc:1875] T e4d8164736d1409a933b06391c27bd8d P 9a4d92bc25374fb3960fde7d10be309d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:05.915596 2063 ts_tablet_manager.cc:1888] T e4d8164736d1409a933b06391c27bd8d P 9a4d92bc25374fb3960fde7d10be309d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:05.915786 2063 log.cc:1196] T e4d8164736d1409a933b06391c27bd8d P 9a4d92bc25374fb3960fde7d10be309d: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal/wals/e4d8164736d1409a933b06391c27bd8d
I20240503 02:13:05.916258 2063 ts_tablet_manager.cc:1909] T e4d8164736d1409a933b06391c27bd8d P 9a4d92bc25374fb3960fde7d10be309d: Deleting consensus metadata
I20240503 02:13:05.917409 29719 catalog_manager.cc:4830] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): tablet e4d8164736d1409a933b06391c27bd8d (table rebalancer_dynamic_table_new_name [id=11c3a184b65d48b6b02c35ba6d559c83]) successfully deleted
I20240503 02:13:05.959862 29731 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:13:05.972021 547 tablet_service.cc:1467] Processing CreateTablet for tablet 1a397174721745f4a7c60f51e893f33a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=316e30bcbc7142fea5235096f42d3e11]), partition=
I20240503 02:13:05.972457 547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a397174721745f4a7c60f51e893f33a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:05.973062 29853 tablet_service.cc:1467] Processing CreateTablet for tablet 1a397174721745f4a7c60f51e893f33a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=316e30bcbc7142fea5235096f42d3e11]), partition=
I20240503 02:13:05.973534 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a397174721745f4a7c60f51e893f33a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:05.977489 32620 tablet_service.cc:1467] Processing CreateTablet for tablet 1a397174721745f4a7c60f51e893f33a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=316e30bcbc7142fea5235096f42d3e11]), partition=
I20240503 02:13:05.977980 32620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a397174721745f4a7c60f51e893f33a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:05.995903 1992 tablet_bootstrap.cc:492] T 1a397174721745f4a7c60f51e893f33a P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:05.998380 1992 tablet_bootstrap.cc:654] T 1a397174721745f4a7c60f51e893f33a P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:06.002241 2079 tablet_bootstrap.cc:492] T 1a397174721745f4a7c60f51e893f33a P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:13:06.006187 2079 tablet_bootstrap.cc:654] T 1a397174721745f4a7c60f51e893f33a P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:06.008577 2079 tablet_bootstrap.cc:492] T 1a397174721745f4a7c60f51e893f33a P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:13:06.009112 2078 tablet_bootstrap.cc:492] T 1a397174721745f4a7c60f51e893f33a P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:06.009367 2079 ts_tablet_manager.cc:1367] T 1a397174721745f4a7c60f51e893f33a P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 02:13:06.010231 2079 raft_consensus.cc:348] T 1a397174721745f4a7c60f51e893f33a P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:06.011394 2079 raft_consensus.cc:374] T 1a397174721745f4a7c60f51e893f33a P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:06.011581 2079 raft_consensus.cc:729] T 1a397174721745f4a7c60f51e893f33a P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:13:06.011946 2079 consensus_queue.cc:260] T 1a397174721745f4a7c60f51e893f33a P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:06.012787 2079 ts_tablet_manager.cc:1398] T 1a397174721745f4a7c60f51e893f33a P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:06.012874 1992 tablet_bootstrap.cc:492] T 1a397174721745f4a7c60f51e893f33a P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:06.014420 2078 tablet_bootstrap.cc:654] T 1a397174721745f4a7c60f51e893f33a P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:06.013146 1992 ts_tablet_manager.cc:1367] T 1a397174721745f4a7c60f51e893f33a P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.000s
I20240503 02:13:06.022791 1992 raft_consensus.cc:348] T 1a397174721745f4a7c60f51e893f33a P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:06.024492 1992 raft_consensus.cc:374] T 1a397174721745f4a7c60f51e893f33a P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:06.024657 1992 raft_consensus.cc:729] T 1a397174721745f4a7c60f51e893f33a P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:06.025049 1992 consensus_queue.cc:260] T 1a397174721745f4a7c60f51e893f33a P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:06.026163 1992 ts_tablet_manager.cc:1398] T 1a397174721745f4a7c60f51e893f33a P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 02:13:06.031301 2078 tablet_bootstrap.cc:492] T 1a397174721745f4a7c60f51e893f33a P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:06.031973 2078 ts_tablet_manager.cc:1367] T 1a397174721745f4a7c60f51e893f33a P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.006s
I20240503 02:13:06.032829 2078 raft_consensus.cc:348] T 1a397174721745f4a7c60f51e893f33a P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:06.033128 2078 raft_consensus.cc:374] T 1a397174721745f4a7c60f51e893f33a P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:06.033255 2078 raft_consensus.cc:729] T 1a397174721745f4a7c60f51e893f33a P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:06.033558 2078 consensus_queue.cc:260] T 1a397174721745f4a7c60f51e893f33a P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:06.037458 2078 ts_tablet_manager.cc:1398] T 1a397174721745f4a7c60f51e893f33a P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:13:06.115973 2018 consensus_queue.cc:1035] T c42a89fd9daf427bb5a60b4ee76a9287 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:06.135684 1976 consensus_queue.cc:1035] T c42a89fd9daf427bb5a60b4ee76a9287 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:06.144105 2025 consensus_queue.cc:1035] T 2a132e4747824d5989858e649bc243c2 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:06.158403 1484 consensus_queue.cc:1035] T c97c7a843366426887115e13f35faf52 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:06.164511 29853 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:06.171851 1484 consensus_queue.cc:1035] T c97c7a843366426887115e13f35faf52 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:06.171973 2025 consensus_queue.cc:1035] T 2a132e4747824d5989858e649bc243c2 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:06.203487 30116 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:06.218969 32620 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:13:06.228121 417 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:06.256790 29725 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_extra_table_20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:06.267774 547 tablet_service.cc:1467] Processing CreateTablet for tablet 9c6e204df01e450598b05c3429574700 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=9fae481dc13a491db2520bd2d31be634]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:06.268268 547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c6e204df01e450598b05c3429574700. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:06.269891 417 tablet_service.cc:1467] Processing CreateTablet for tablet 9c6e204df01e450598b05c3429574700 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=9fae481dc13a491db2520bd2d31be634]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:06.270123 32620 tablet_service.cc:1467] Processing CreateTablet for tablet 36d3f1edbab7477f9b7ac400160491a0 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=9fae481dc13a491db2520bd2d31be634]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:06.270437 417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c6e204df01e450598b05c3429574700. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:06.270530 32620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36d3f1edbab7477f9b7ac400160491a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:06.273483 29853 tablet_service.cc:1467] Processing CreateTablet for tablet 75497fcddb62421395bd0fa7c88a406c (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=9fae481dc13a491db2520bd2d31be634]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:06.273931 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75497fcddb62421395bd0fa7c88a406c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:06.276767 32619 tablet_service.cc:1467] Processing CreateTablet for tablet 75497fcddb62421395bd0fa7c88a406c (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=9fae481dc13a491db2520bd2d31be634]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:06.277374 32619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75497fcddb62421395bd0fa7c88a406c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:06.277608 30116 tablet_service.cc:1467] Processing CreateTablet for tablet 36d3f1edbab7477f9b7ac400160491a0 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=9fae481dc13a491db2520bd2d31be634]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:06.278029 30116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36d3f1edbab7477f9b7ac400160491a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:06.280732 1992 tablet_bootstrap.cc:492] T 36d3f1edbab7477f9b7ac400160491a0 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:06.281881 546 tablet_service.cc:1467] Processing CreateTablet for tablet 36d3f1edbab7477f9b7ac400160491a0 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=9fae481dc13a491db2520bd2d31be634]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:06.282047 548 tablet_service.cc:1467] Processing CreateTablet for tablet 75497fcddb62421395bd0fa7c88a406c (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=9fae481dc13a491db2520bd2d31be634]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:06.282322 546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36d3f1edbab7477f9b7ac400160491a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:06.282500 1992 tablet_bootstrap.cc:654] T 36d3f1edbab7477f9b7ac400160491a0 P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:06.284209 548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75497fcddb62421395bd0fa7c88a406c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:06.285701 2025 raft_consensus.cc:482] T 1a397174721745f4a7c60f51e893f33a P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:06.285851 2025 raft_consensus.cc:504] T 1a397174721745f4a7c60f51e893f33a P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:06.286726 2025 leader_election.cc:288] T 1a397174721745f4a7c60f51e893f33a P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:13:06.289337 2078 tablet_bootstrap.cc:492] T 36d3f1edbab7477f9b7ac400160491a0 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:06.289428 566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a397174721745f4a7c60f51e893f33a" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:06.289788 566 raft_consensus.cc:2454] T 1a397174721745f4a7c60f51e893f33a P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 0.
I20240503 02:13:06.291610 32573 leader_election.cc:302] T 1a397174721745f4a7c60f51e893f33a P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:06.291919 2025 raft_consensus.cc:2781] T 1a397174721745f4a7c60f51e893f33a P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:06.292034 2025 raft_consensus.cc:482] T 1a397174721745f4a7c60f51e893f33a P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:06.292102 2078 tablet_bootstrap.cc:654] T 36d3f1edbab7477f9b7ac400160491a0 P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:06.292121 2025 raft_consensus.cc:3037] T 1a397174721745f4a7c60f51e893f33a P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:06.295832 2025 raft_consensus.cc:504] T 1a397174721745f4a7c60f51e893f33a P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:06.298365 566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a397174721745f4a7c60f51e893f33a" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:13:06.298624 566 raft_consensus.cc:3037] T 1a397174721745f4a7c60f51e893f33a P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:06.302680 566 raft_consensus.cc:2454] T 1a397174721745f4a7c60f51e893f33a P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 1.
I20240503 02:13:06.303288 32573 leader_election.cc:302] T 1a397174721745f4a7c60f51e893f33a P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:06.303628 1787 raft_consensus.cc:2781] T 1a397174721745f4a7c60f51e893f33a P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:06.303803 1787 raft_consensus.cc:686] T 1a397174721745f4a7c60f51e893f33a P 9a4d92bc25374fb3960fde7d10be309d [term 1 LEADER]: Becoming Leader. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Running, Role: LEADER
I20240503 02:13:06.304095 1787 consensus_queue.cc:237] T 1a397174721745f4a7c60f51e893f33a P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:06.310966 2079 tablet_bootstrap.cc:492] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:13:06.313328 2079 tablet_bootstrap.cc:654] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:06.314978 2025 leader_election.cc:288] T 1a397174721745f4a7c60f51e893f33a P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 election: Requested vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:13:06.317582 2078 tablet_bootstrap.cc:492] T 36d3f1edbab7477f9b7ac400160491a0 P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:06.318003 2078 ts_tablet_manager.cc:1367] T 36d3f1edbab7477f9b7ac400160491a0 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.004s
I20240503 02:13:06.319108 2078 raft_consensus.cc:348] T 36d3f1edbab7477f9b7ac400160491a0 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:06.319578 2078 raft_consensus.cc:374] T 36d3f1edbab7477f9b7ac400160491a0 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:06.319716 2078 raft_consensus.cc:729] T 36d3f1edbab7477f9b7ac400160491a0 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:06.320039 2078 consensus_queue.cc:260] T 36d3f1edbab7477f9b7ac400160491a0 P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:06.321517 2078 ts_tablet_manager.cc:1398] T 36d3f1edbab7477f9b7ac400160491a0 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:06.325971 2078 tablet_bootstrap.cc:492] T 75497fcddb62421395bd0fa7c88a406c P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:06.329599 29733 catalog_manager.cc:5484] T 1a397174721745f4a7c60f51e893f33a P 9a4d92bc25374fb3960fde7d10be309d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5). New cstate: current_term: 1 leader_uuid: "9a4d92bc25374fb3960fde7d10be309d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:06.333835 1992 tablet_bootstrap.cc:492] T 36d3f1edbab7477f9b7ac400160491a0 P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:06.334053 1992 ts_tablet_manager.cc:1367] T 36d3f1edbab7477f9b7ac400160491a0 P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.053s user 0.008s sys 0.000s
I20240503 02:13:06.335088 1992 raft_consensus.cc:348] T 36d3f1edbab7477f9b7ac400160491a0 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:06.335691 1992 raft_consensus.cc:374] T 36d3f1edbab7477f9b7ac400160491a0 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:06.336475 2079 tablet_bootstrap.cc:492] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:13:06.336730 2079 ts_tablet_manager.cc:1367] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.009s
I20240503 02:13:06.337589 2079 raft_consensus.cc:348] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:06.337895 2079 raft_consensus.cc:374] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:06.338034 2079 raft_consensus.cc:729] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:13:06.339488 2078 tablet_bootstrap.cc:654] T 75497fcddb62421395bd0fa7c88a406c P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:06.340055 2079 consensus_queue.cc:260] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:06.341078 2079 ts_tablet_manager.cc:1398] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:13:06.341382 1992 raft_consensus.cc:729] T 36d3f1edbab7477f9b7ac400160491a0 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:06.341825 1992 consensus_queue.cc:260] T 36d3f1edbab7477f9b7ac400160491a0 P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:06.343044 2126 tablet_bootstrap.cc:492] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:13:06.345046 1992 ts_tablet_manager.cc:1398] T 36d3f1edbab7477f9b7ac400160491a0 P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 02:13:06.345443 1992 tablet_bootstrap.cc:492] T 75497fcddb62421395bd0fa7c88a406c P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:06.349094 1992 tablet_bootstrap.cc:654] T 75497fcddb62421395bd0fa7c88a406c P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:06.361853 1484 raft_consensus.cc:482] T 1a397174721745f4a7c60f51e893f33a P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:06.362063 1484 raft_consensus.cc:504] T 1a397174721745f4a7c60f51e893f33a P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:06.362833 1484 leader_election.cc:288] T 1a397174721745f4a7c60f51e893f33a P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:13:06.363621 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a397174721745f4a7c60f51e893f33a" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:13:06.367636 566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a397174721745f4a7c60f51e893f33a" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:06.368470 566 raft_consensus.cc:2379] T 1a397174721745f4a7c60f51e893f33a P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 16440e365430405fb10366ab70e2b429 in current term 1: Already voted for candidate 9a4d92bc25374fb3960fde7d10be309d in this term.
I20240503 02:13:06.369191 29808 leader_election.cc:302] T 1a397174721745f4a7c60f51e893f33a P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 16440e365430405fb10366ab70e2b429; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:13:06.369894 1484 raft_consensus.cc:3037] T 1a397174721745f4a7c60f51e893f33a P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:06.370725 2078 tablet_bootstrap.cc:492] T 75497fcddb62421395bd0fa7c88a406c P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:06.370934 2078 ts_tablet_manager.cc:1367] T 75497fcddb62421395bd0fa7c88a406c P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.003s
I20240503 02:13:06.372207 2078 raft_consensus.cc:348] T 75497fcddb62421395bd0fa7c88a406c P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:06.372457 2078 raft_consensus.cc:374] T 75497fcddb62421395bd0fa7c88a406c P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:06.372573 2078 raft_consensus.cc:729] T 75497fcddb62421395bd0fa7c88a406c P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:06.374392 1484 raft_consensus.cc:2726] T 1a397174721745f4a7c60f51e893f33a P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:06.375602 2078 consensus_queue.cc:260] T 75497fcddb62421395bd0fa7c88a406c P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:06.376324 2126 tablet_bootstrap.cc:654] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:06.376868 2078 ts_tablet_manager.cc:1398] T 75497fcddb62421395bd0fa7c88a406c P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.006s user 0.002s sys 0.002s
I20240503 02:13:06.377199 2078 tablet_bootstrap.cc:492] T 9c6e204df01e450598b05c3429574700 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:06.381326 2078 tablet_bootstrap.cc:654] T 9c6e204df01e450598b05c3429574700 P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:06.392019 2078 tablet_bootstrap.cc:492] T 9c6e204df01e450598b05c3429574700 P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:06.393072 2078 ts_tablet_manager.cc:1367] T 9c6e204df01e450598b05c3429574700 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.006s
I20240503 02:13:06.393440 32755 tablet_service.cc:1467] Processing CreateTablet for tablet 9c6e204df01e450598b05c3429574700 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=9fae481dc13a491db2520bd2d31be634]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:06.394016 32755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c6e204df01e450598b05c3429574700. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:06.396867 2078 raft_consensus.cc:348] T 9c6e204df01e450598b05c3429574700 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:06.397192 2078 raft_consensus.cc:374] T 9c6e204df01e450598b05c3429574700 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:06.397284 2078 raft_consensus.cc:729] T 9c6e204df01e450598b05c3429574700 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:06.398908 2078 consensus_queue.cc:260] T 9c6e204df01e450598b05c3429574700 P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:06.401073 2078 ts_tablet_manager.cc:1398] T 9c6e204df01e450598b05c3429574700 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240503 02:13:06.410773 2126 tablet_bootstrap.cc:492] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:13:06.411737 2126 ts_tablet_manager.cc:1367] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.069s user 0.008s sys 0.004s
I20240503 02:13:06.414233 2126 raft_consensus.cc:348] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:06.414610 2126 raft_consensus.cc:374] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:06.414822 2126 raft_consensus.cc:729] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:13:06.415174 2126 consensus_queue.cc:260] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:06.416121 2126 ts_tablet_manager.cc:1398] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:13:06.437100 1909 raft_consensus.cc:482] T 36d3f1edbab7477f9b7ac400160491a0 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:06.437395 1909 raft_consensus.cc:504] T 36d3f1edbab7477f9b7ac400160491a0 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:06.438661 1909 leader_election.cc:288] T 36d3f1edbab7477f9b7ac400160491a0 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:13:06.440114 30135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d3f1edbab7477f9b7ac400160491a0" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:13:06.440321 1787 raft_consensus.cc:482] T 36d3f1edbab7477f9b7ac400160491a0 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:06.440399 30135 raft_consensus.cc:2454] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 0.
I20240503 02:13:06.440508 1787 raft_consensus.cc:504] T 36d3f1edbab7477f9b7ac400160491a0 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:06.441195 501 leader_election.cc:302] T 36d3f1edbab7477f9b7ac400160491a0 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:06.441344 1787 leader_election.cc:288] T 36d3f1edbab7477f9b7ac400160491a0 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:13:06.441759 1909 raft_consensus.cc:2781] T 36d3f1edbab7477f9b7ac400160491a0 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:06.441941 1909 raft_consensus.cc:482] T 36d3f1edbab7477f9b7ac400160491a0 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:06.442175 1909 raft_consensus.cc:3037] T 36d3f1edbab7477f9b7ac400160491a0 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:06.443089 30135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d3f1edbab7477f9b7ac400160491a0" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:13:06.443423 30135 raft_consensus.cc:2454] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 0.
I20240503 02:13:06.444028 32573 leader_election.cc:302] T 36d3f1edbab7477f9b7ac400160491a0 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:13:06.444818 1787 raft_consensus.cc:2781] T 36d3f1edbab7477f9b7ac400160491a0 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:06.444957 1787 raft_consensus.cc:482] T 36d3f1edbab7477f9b7ac400160491a0 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:06.445156 1787 raft_consensus.cc:3037] T 36d3f1edbab7477f9b7ac400160491a0 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:06.447489 1909 raft_consensus.cc:504] T 36d3f1edbab7477f9b7ac400160491a0 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:06.449280 30135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d3f1edbab7477f9b7ac400160491a0" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:13:06.449590 30135 raft_consensus.cc:3037] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:06.451345 1787 raft_consensus.cc:504] T 36d3f1edbab7477f9b7ac400160491a0 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:06.452710 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d3f1edbab7477f9b7ac400160491a0" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:13:06.452752 566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d3f1edbab7477f9b7ac400160491a0" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:06.453065 567 raft_consensus.cc:2379] T 36d3f1edbab7477f9b7ac400160491a0 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a4d92bc25374fb3960fde7d10be309d in current term 1: Already voted for candidate f575ce92c0224eeb80b9fae5f6561fbe in this term.
I20240503 02:13:06.453666 30135 raft_consensus.cc:2454] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 1.
I20240503 02:13:06.454247 501 leader_election.cc:302] T 36d3f1edbab7477f9b7ac400160491a0 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:06.454658 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d3f1edbab7477f9b7ac400160491a0" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:13:06.454985 30134 raft_consensus.cc:2379] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a4d92bc25374fb3960fde7d10be309d in current term 1: Already voted for candidate f575ce92c0224eeb80b9fae5f6561fbe in this term.
I20240503 02:13:06.455173 1909 leader_election.cc:288] T 36d3f1edbab7477f9b7ac400160491a0 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 election: Requested vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:13:06.455588 32573 leader_election.cc:302] T 36d3f1edbab7477f9b7ac400160491a0 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d; no voters: a9094240860e43bfb129193f3bfd5be6, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:13:06.455219 2134 raft_consensus.cc:2781] T 36d3f1edbab7477f9b7ac400160491a0 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:06.456076 1787 leader_election.cc:288] T 36d3f1edbab7477f9b7ac400160491a0 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 election: Requested vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:13:06.456535 1787 raft_consensus.cc:2726] T 36d3f1edbab7477f9b7ac400160491a0 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 02:13:06.456508 2134 raft_consensus.cc:686] T 36d3f1edbab7477f9b7ac400160491a0 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 LEADER]: Becoming Leader. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Running, Role: LEADER
I20240503 02:13:06.456969 2134 consensus_queue.cc:237] T 36d3f1edbab7477f9b7ac400160491a0 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:06.477458 29732 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:13:06.535028 566 raft_consensus.cc:1261] T 1a397174721745f4a7c60f51e893f33a P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:06.536163 1787 consensus_queue.cc:1035] T 1a397174721745f4a7c60f51e893f33a P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:06.559811 2141 tablet.cc:1722] T 1a397174721745f4a7c60f51e893f33a P 9a4d92bc25374fb3960fde7d10be309d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:06.562685 2141 tablet.cc:1620] T 1a397174721745f4a7c60f51e893f33a P 9a4d92bc25374fb3960fde7d10be309d: MemRowSet was empty: no flush needed.
I20240503 02:13:06.563252 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d3f1edbab7477f9b7ac400160491a0" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:13:06.563606 32640 raft_consensus.cc:2379] T 36d3f1edbab7477f9b7ac400160491a0 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f575ce92c0224eeb80b9fae5f6561fbe in current term 1: Already voted for candidate 9a4d92bc25374fb3960fde7d10be309d in this term.
I20240503 02:13:06.564299 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d3f1edbab7477f9b7ac400160491a0" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d"
I20240503 02:13:06.564574 32640 raft_consensus.cc:2379] T 36d3f1edbab7477f9b7ac400160491a0 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f575ce92c0224eeb80b9fae5f6561fbe in current term 1: Already voted for candidate 9a4d92bc25374fb3960fde7d10be309d in this term.
I20240503 02:13:06.570139 2124 tablet_bootstrap.cc:492] T 9c6e204df01e450598b05c3429574700 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:06.575171 2124 tablet_bootstrap.cc:654] T 9c6e204df01e450598b05c3429574700 P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:06.576035 2132 tablet_bootstrap.cc:492] T 9c6e204df01e450598b05c3429574700 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:06.582465 2142 tablet.cc:1722] T 1a397174721745f4a7c60f51e893f33a P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:06.583083 2142 tablet.cc:1620] T 1a397174721745f4a7c60f51e893f33a P f575ce92c0224eeb80b9fae5f6561fbe: MemRowSet was empty: no flush needed.
I20240503 02:13:06.584844 29732 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=316e30bcbc7142fea5235096f42d3e11] alter complete (version 1)
I20240503 02:13:06.586086 29986 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240503 02:13:06.595357 30199 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.3:42679, user_credentials={real_user=slave}} blocked reactor thread for 176328us
I20240503 02:13:06.608423 30245 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:06.610965 2136 raft_consensus.cc:482] T 9c6e204df01e450598b05c3429574700 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:06.611202 2136 raft_consensus.cc:504] T 9c6e204df01e450598b05c3429574700 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:06.612316 2136 leader_election.cc:288] T 9c6e204df01e450598b05c3429574700 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:06.613941 2027 raft_consensus.cc:482] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:06.614104 2027 raft_consensus.cc:504] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:06.614801 2027 leader_election.cc:288] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:13:06.615604 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75497fcddb62421395bd0fa7c88a406c" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:06.615854 567 raft_consensus.cc:2454] T 75497fcddb62421395bd0fa7c88a406c P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:13:06.615942 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75497fcddb62421395bd0fa7c88a406c" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
W20240503 02:13:06.616580 29809 leader_election.cc:341] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:06.617353 29808 leader_election.cc:302] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16440e365430405fb10366ab70e2b429, f575ce92c0224eeb80b9fae5f6561fbe; no voters: 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:13:06.617794 2027 raft_consensus.cc:2781] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:06.617909 2027 raft_consensus.cc:482] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:06.618007 2027 raft_consensus.cc:3037] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:06.620764 2027 raft_consensus.cc:504] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:06.622856 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75497fcddb62421395bd0fa7c88a406c" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:13:06.623108 567 raft_consensus.cc:3037] T 75497fcddb62421395bd0fa7c88a406c P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:06.624352 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75497fcddb62421395bd0fa7c88a406c" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d"
W20240503 02:13:06.624939 29809 leader_election.cc:341] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:06.625134 2027 leader_election.cc:288] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Requested vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:13:06.633919 567 raft_consensus.cc:2454] T 75497fcddb62421395bd0fa7c88a406c P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:13:06.634613 29808 leader_election.cc:302] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16440e365430405fb10366ab70e2b429, f575ce92c0224eeb80b9fae5f6561fbe; no voters: 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:13:06.635083 2027 raft_consensus.cc:2781] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:06.635224 2027 raft_consensus.cc:686] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Becoming Leader. State: Replica: 16440e365430405fb10366ab70e2b429, State: Running, Role: LEADER
I20240503 02:13:06.635538 2027 consensus_queue.cc:237] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:06.637907 2132 tablet_bootstrap.cc:654] T 9c6e204df01e450598b05c3429574700 P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:06.640524 29725 catalog_manager.cc:5484] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16440e365430405fb10366ab70e2b429 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:06.652586 32755 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:13:06.653620 502 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.6:39975, user_credentials={real_user=slave}} blocked reactor thread for 187215us
I20240503 02:13:06.656167 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6e204df01e450598b05c3429574700" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
W20240503 02:13:06.662796 502 leader_election.cc:341] T 9c6e204df01e450598b05c3429574700 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:06.657630 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6e204df01e450598b05c3429574700" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
W20240503 02:13:06.668323 502 leader_election.cc:341] T 9c6e204df01e450598b05c3429574700 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:06.668612 502 leader_election.cc:302] T 9c6e204df01e450598b05c3429574700 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f575ce92c0224eeb80b9fae5f6561fbe; no voters: 8a666e798fe8477d8dd1e51b385d5ead, fbbc2be6382e47908ecb7c68c367bd26
I20240503 02:13:06.668931 1909 raft_consensus.cc:2726] T 9c6e204df01e450598b05c3429574700 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:06.682260 29725 catalog_manager.cc:5484] T 36d3f1edbab7477f9b7ac400160491a0 P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: term changed from 0 to 1, leader changed from <none> to f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8). New cstate: current_term: 1 leader_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:06.689524 547 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:13:06.698694 29874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a397174721745f4a7c60f51e893f33a" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:13:06.698968 29874 raft_consensus.cc:2454] T 1a397174721745f4a7c60f51e893f33a P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 1.
I20240503 02:13:06.699563 29874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a397174721745f4a7c60f51e893f33a" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:13:06.702415 29874 raft_consensus.cc:2454] T 1a397174721745f4a7c60f51e893f33a P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 1.
I20240503 02:13:06.703718 29874 raft_consensus.cc:1261] T 1a397174721745f4a7c60f51e893f33a P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:06.704334 2025 consensus_queue.cc:1035] T 1a397174721745f4a7c60f51e893f33a P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:06.714641 29733 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:13:06.726171 2141 tablet.cc:1722] T 1a397174721745f4a7c60f51e893f33a P 9a4d92bc25374fb3960fde7d10be309d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:06.726677 2141 tablet.cc:1620] T 1a397174721745f4a7c60f51e893f33a P 9a4d92bc25374fb3960fde7d10be309d: MemRowSet was empty: no flush needed.
I20240503 02:13:06.726944 2124 tablet_bootstrap.cc:492] T 9c6e204df01e450598b05c3429574700 P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:06.728199 2142 tablet.cc:1722] T 1a397174721745f4a7c60f51e893f33a P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:06.728761 2142 tablet.cc:1620] T 1a397174721745f4a7c60f51e893f33a P f575ce92c0224eeb80b9fae5f6561fbe: MemRowSet was empty: no flush needed.
I20240503 02:13:06.729552 2132 tablet_bootstrap.cc:492] T 9c6e204df01e450598b05c3429574700 P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:06.729804 2132 ts_tablet_manager.cc:1367] T 9c6e204df01e450598b05c3429574700 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.154s user 0.006s sys 0.000s
I20240503 02:13:06.730683 2132 raft_consensus.cc:348] T 9c6e204df01e450598b05c3429574700 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:06.731988 2124 ts_tablet_manager.cc:1367] T 9c6e204df01e450598b05c3429574700 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.162s user 0.005s sys 0.001s
I20240503 02:13:06.733103 1992 tablet_bootstrap.cc:492] T 75497fcddb62421395bd0fa7c88a406c P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:06.733383 1992 ts_tablet_manager.cc:1367] T 75497fcddb62421395bd0fa7c88a406c P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.388s user 0.092s sys 0.089s
I20240503 02:13:06.734160 1992 raft_consensus.cc:348] T 75497fcddb62421395bd0fa7c88a406c P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:06.734398 1992 raft_consensus.cc:374] T 75497fcddb62421395bd0fa7c88a406c P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:06.734481 1992 raft_consensus.cc:729] T 75497fcddb62421395bd0fa7c88a406c P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:06.734763 1992 consensus_queue.cc:260] T 75497fcddb62421395bd0fa7c88a406c P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:06.735397 1992 ts_tablet_manager.cc:1398] T 75497fcddb62421395bd0fa7c88a406c P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:06.738193 29733 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=316e30bcbc7142fea5235096f42d3e11] alter complete (version 2)
I20240503 02:13:06.738337 2124 raft_consensus.cc:348] T 9c6e204df01e450598b05c3429574700 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:06.739953 2149 tablet.cc:1722] T 1a397174721745f4a7c60f51e893f33a P 16440e365430405fb10366ab70e2b429: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:06.740943 2149 tablet.cc:1620] T 1a397174721745f4a7c60f51e893f33a P 16440e365430405fb10366ab70e2b429: MemRowSet was empty: no flush needed.
I20240503 02:13:06.742693 2132 raft_consensus.cc:374] T 9c6e204df01e450598b05c3429574700 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:06.743168 2124 raft_consensus.cc:374] T 9c6e204df01e450598b05c3429574700 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:06.743347 2132 raft_consensus.cc:729] T 9c6e204df01e450598b05c3429574700 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:06.744599 2124 raft_consensus.cc:729] T 9c6e204df01e450598b05c3429574700 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:06.747215 29725 catalog_manager.cc:5576] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719) does not have the latest schema for tablet 1a397174721745f4a7c60f51e893f33a (table rebalancer_dynamic_table [id=316e30bcbc7142fea5235096f42d3e11]). Expected version 2 got 0
I20240503 02:13:06.749204 2132 consensus_queue.cc:260] T 9c6e204df01e450598b05c3429574700 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:06.750319 2124 consensus_queue.cc:260] T 9c6e204df01e450598b05c3429574700 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:06.750972 2124 ts_tablet_manager.cc:1398] T 9c6e204df01e450598b05c3429574700 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.019s user 0.000s sys 0.001s
I20240503 02:13:06.750999 2132 ts_tablet_manager.cc:1398] T 9c6e204df01e450598b05c3429574700 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20240503 02:13:06.753330 29732 catalog_manager.cc:5576] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719) does not have the latest schema for tablet 1a397174721745f4a7c60f51e893f33a (table rebalancer_dynamic_table [id=316e30bcbc7142fea5235096f42d3e11]). Expected version 2 got 1
I20240503 02:13:06.759902 2149 tablet.cc:1722] T 1a397174721745f4a7c60f51e893f33a P 16440e365430405fb10366ab70e2b429: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:06.760303 2149 tablet.cc:1620] T 1a397174721745f4a7c60f51e893f33a P 16440e365430405fb10366ab70e2b429: MemRowSet was empty: no flush needed.
I20240503 02:13:06.847031 29733 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:13:06.860134 2141 tablet.cc:1722] T 1a397174721745f4a7c60f51e893f33a P 9a4d92bc25374fb3960fde7d10be309d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:06.861589 2142 tablet.cc:1722] T 1a397174721745f4a7c60f51e893f33a P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:06.866312 2149 tablet.cc:1722] T 1a397174721745f4a7c60f51e893f33a P 16440e365430405fb10366ab70e2b429: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:06.883847 29732 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table_new_name [id=316e30bcbc7142fea5235096f42d3e11] alter complete (version 3)
I20240503 02:13:06.907498 2134 consensus_queue.cc:1035] T 36d3f1edbab7477f9b7ac400160491a0 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:06.914057 2136 consensus_queue.cc:1035] T 36d3f1edbab7477f9b7ac400160491a0 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:06.992657 1900 raft_consensus.cc:482] T 9c6e204df01e450598b05c3429574700 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:06.992849 1900 raft_consensus.cc:504] T 9c6e204df01e450598b05c3429574700 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:06.996057 1900 leader_election.cc:288] T 9c6e204df01e450598b05c3429574700 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:06.996786 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6e204df01e450598b05c3429574700" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:06.996831 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6e204df01e450598b05c3429574700" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:13:06.997013 567 raft_consensus.cc:2454] T 9c6e204df01e450598b05c3429574700 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 0.
I20240503 02:13:06.997112 307 raft_consensus.cc:2454] T 9c6e204df01e450598b05c3429574700 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 0.
I20240503 02:13:06.997756 371 leader_election.cc:302] T 9c6e204df01e450598b05c3429574700 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:06.998404 2094 raft_consensus.cc:2781] T 9c6e204df01e450598b05c3429574700 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:06.998499 2094 raft_consensus.cc:482] T 9c6e204df01e450598b05c3429574700 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:06.998607 2094 raft_consensus.cc:3037] T 9c6e204df01e450598b05c3429574700 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:06.998759 29732 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:06.998965 29732 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:07.001403 2094 raft_consensus.cc:504] T 9c6e204df01e450598b05c3429574700 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:07.002193 2094 leader_election.cc:288] T 9c6e204df01e450598b05c3429574700 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Requested vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:07.002415 29732 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet 1a397174721745f4a7c60f51e893f33a
I20240503 02:13:07.003237 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6e204df01e450598b05c3429574700" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:13:07.003479 307 raft_consensus.cc:3037] T 9c6e204df01e450598b05c3429574700 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:07.003561 32620 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a397174721745f4a7c60f51e893f33a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:06 UTC) from {username='slave'} at 127.0.0.1:38200
I20240503 02:13:07.005556 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c6e204df01e450598b05c3429574700" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:13:07.005769 567 raft_consensus.cc:3037] T 9c6e204df01e450598b05c3429574700 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:07.007207 307 raft_consensus.cc:2454] T 9c6e204df01e450598b05c3429574700 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 1.
I20240503 02:13:07.007691 29853 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a397174721745f4a7c60f51e893f33a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:06 UTC) from {username='slave'} at 127.0.0.1:58744
I20240503 02:13:07.007989 371 leader_election.cc:302] T 9c6e204df01e450598b05c3429574700 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:07.008284 547 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a397174721745f4a7c60f51e893f33a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:06 UTC) from {username='slave'} at 127.0.0.1:39028
I20240503 02:13:07.008405 1900 raft_consensus.cc:2781] T 9c6e204df01e450598b05c3429574700 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:07.009506 567 raft_consensus.cc:2454] T 9c6e204df01e450598b05c3429574700 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 1.
I20240503 02:13:07.014303 30266 consensus_queue.cc:237] T 96de1c26ae884ea18b63b328db040ba1 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:07.018033 2174 tablet_replica.cc:330] T 1a397174721745f4a7c60f51e893f33a P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:13:07.018324 2174 raft_consensus.cc:2229] T 1a397174721745f4a7c60f51e893f33a P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:07.018611 2174 raft_consensus.cc:2258] T 1a397174721745f4a7c60f51e893f33a P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:07.031776 2175 tablet_replica.cc:330] T 1a397174721745f4a7c60f51e893f33a P f575ce92c0224eeb80b9fae5f6561fbe: stopping tablet replica
I20240503 02:13:07.032117 2175 raft_consensus.cc:2229] T 1a397174721745f4a7c60f51e893f33a P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:07.032423 2175 raft_consensus.cc:2258] T 1a397174721745f4a7c60f51e893f33a P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:07.032696 30135 raft_consensus.cc:1261] T 96de1c26ae884ea18b63b328db040ba1 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:07.032728 1900 raft_consensus.cc:686] T 9c6e204df01e450598b05c3429574700 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Becoming Leader. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Running, Role: LEADER
I20240503 02:13:07.033138 1900 consensus_queue.cc:237] T 9c6e204df01e450598b05c3429574700 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:07.033391 437 raft_consensus.cc:1261] T 96de1c26ae884ea18b63b328db040ba1 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:07.033480 1533 consensus_queue.cc:1035] T 96de1c26ae884ea18b63b328db040ba1 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:07.034314 1533 consensus_queue.cc:1035] T 96de1c26ae884ea18b63b328db040ba1 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:07.034752 2173 tablet_replica.cc:330] T 1a397174721745f4a7c60f51e893f33a P 9a4d92bc25374fb3960fde7d10be309d: stopping tablet replica
I20240503 02:13:07.034868 2175 ts_tablet_manager.cc:1875] T 1a397174721745f4a7c60f51e893f33a P f575ce92c0224eeb80b9fae5f6561fbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:07.034971 2173 raft_consensus.cc:2229] T 1a397174721745f4a7c60f51e893f33a P 9a4d92bc25374fb3960fde7d10be309d [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:07.035833 2173 raft_consensus.cc:2258] T 1a397174721745f4a7c60f51e893f33a P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:07.037286 2173 ts_tablet_manager.cc:1875] T 1a397174721745f4a7c60f51e893f33a P 9a4d92bc25374fb3960fde7d10be309d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:07.043498 2175 ts_tablet_manager.cc:1888] T 1a397174721745f4a7c60f51e893f33a P f575ce92c0224eeb80b9fae5f6561fbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:07.043776 2175 log.cc:1196] T 1a397174721745f4a7c60f51e893f33a P f575ce92c0224eeb80b9fae5f6561fbe: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal/wals/1a397174721745f4a7c60f51e893f33a
I20240503 02:13:07.044413 2175 ts_tablet_manager.cc:1909] T 1a397174721745f4a7c60f51e893f33a P f575ce92c0224eeb80b9fae5f6561fbe: Deleting consensus metadata
I20240503 02:13:07.049877 29718 catalog_manager.cc:4830] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): tablet 1a397174721745f4a7c60f51e893f33a (table rebalancer_dynamic_table_new_name [id=316e30bcbc7142fea5235096f42d3e11]) successfully deleted
I20240503 02:13:07.051673 2174 ts_tablet_manager.cc:1875] T 1a397174721745f4a7c60f51e893f33a P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:07.051522 2133 raft_consensus.cc:2932] T 96de1c26ae884ea18b63b328db040ba1 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:07.052618 2173 ts_tablet_manager.cc:1888] T 1a397174721745f4a7c60f51e893f33a P 9a4d92bc25374fb3960fde7d10be309d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:07.052744 2173 log.cc:1196] T 1a397174721745f4a7c60f51e893f33a P 9a4d92bc25374fb3960fde7d10be309d: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal/wals/1a397174721745f4a7c60f51e893f33a
I20240503 02:13:07.053325 2173 ts_tablet_manager.cc:1909] T 1a397174721745f4a7c60f51e893f33a P 9a4d92bc25374fb3960fde7d10be309d: Deleting consensus metadata
I20240503 02:13:07.055543 29719 catalog_manager.cc:4830] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): tablet 1a397174721745f4a7c60f51e893f33a (table rebalancer_dynamic_table_new_name [id=316e30bcbc7142fea5235096f42d3e11]) successfully deleted
I20240503 02:13:07.055382 437 raft_consensus.cc:2932] T 96de1c26ae884ea18b63b328db040ba1 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:07.059329 2174 ts_tablet_manager.cc:1888] T 1a397174721745f4a7c60f51e893f33a P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:07.059473 2174 log.cc:1196] T 1a397174721745f4a7c60f51e893f33a P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/1a397174721745f4a7c60f51e893f33a
I20240503 02:13:07.060050 2174 ts_tablet_manager.cc:1909] T 1a397174721745f4a7c60f51e893f33a P 16440e365430405fb10366ab70e2b429: Deleting consensus metadata
I20240503 02:13:07.062065 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet 1a397174721745f4a7c60f51e893f33a (table rebalancer_dynamic_table_new_name [id=316e30bcbc7142fea5235096f42d3e11]) successfully deleted
I20240503 02:13:07.063475 29732 catalog_manager.cc:5484] T 96de1c26ae884ea18b63b328db040ba1 P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: config changed from index 4 to 5, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:07.067759 29853 tablet_service.cc:1514] Processing DeleteTablet for tablet 96de1c26ae884ea18b63b328db040ba1 with delete_type TABLET_DATA_TOMBSTONED (TS 16440e365430405fb10366ab70e2b429 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:58744
I20240503 02:13:07.069219 2174 tablet_replica.cc:330] T 96de1c26ae884ea18b63b328db040ba1 P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:13:07.070039 2174 raft_consensus.cc:2229] T 96de1c26ae884ea18b63b328db040ba1 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:07.070348 2174 raft_consensus.cc:2258] T 96de1c26ae884ea18b63b328db040ba1 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:07.070641 1787 raft_consensus.cc:482] T 75497fcddb62421395bd0fa7c88a406c P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:07.070849 1787 raft_consensus.cc:504] T 75497fcddb62421395bd0fa7c88a406c P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:07.071184 2174 ts_tablet_manager.cc:1875] T 96de1c26ae884ea18b63b328db040ba1 P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:07.071163 29732 catalog_manager.cc:5484] T 9c6e204df01e450598b05c3429574700 P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7). New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:07.071516 1787 leader_election.cc:288] T 75497fcddb62421395bd0fa7c88a406c P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:13:07.071941 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75497fcddb62421395bd0fa7c88a406c" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:07.072288 567 raft_consensus.cc:2379] T 75497fcddb62421395bd0fa7c88a406c P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a4d92bc25374fb3960fde7d10be309d in current term 1: Already voted for candidate 16440e365430405fb10366ab70e2b429 in this term.
I20240503 02:13:07.075533 29874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75497fcddb62421395bd0fa7c88a406c" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:13:07.076364 32576 leader_election.cc:302] T 75497fcddb62421395bd0fa7c88a406c P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d; no voters: 16440e365430405fb10366ab70e2b429, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:13:07.077607 1787 raft_consensus.cc:3037] T 75497fcddb62421395bd0fa7c88a406c P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:07.077950 2174 ts_tablet_manager.cc:1888] T 96de1c26ae884ea18b63b328db040ba1 P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:13:07.078099 2174 log.cc:1196] T 96de1c26ae884ea18b63b328db040ba1 P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/96de1c26ae884ea18b63b328db040ba1
I20240503 02:13:07.079360 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet 96de1c26ae884ea18b63b328db040ba1 (table rebalance_test_table_1 [id=28d7eeaed8114076b7afb44e4f89a2ea]) successfully deleted
I20240503 02:13:07.082562 1787 raft_consensus.cc:2726] T 75497fcddb62421395bd0fa7c88a406c P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:07.083051 30135 raft_consensus.cc:2932] T 96de1c26ae884ea18b63b328db040ba1 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:07.108043 29732 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:13:07.120247 417 tablet_service.cc:1467] Processing CreateTablet for tablet 24f36508d09446c2ba7bdcec1fcf72ab (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7026528a8ba24559a989f4be8991ad2c]), partition=
I20240503 02:13:07.120676 417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24f36508d09446c2ba7bdcec1fcf72ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:07.122088 547 tablet_service.cc:1467] Processing CreateTablet for tablet 24f36508d09446c2ba7bdcec1fcf72ab (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7026528a8ba24559a989f4be8991ad2c]), partition=
I20240503 02:13:07.122502 547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24f36508d09446c2ba7bdcec1fcf72ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:07.122499 32755 tablet_service.cc:1467] Processing CreateTablet for tablet 24f36508d09446c2ba7bdcec1fcf72ab (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7026528a8ba24559a989f4be8991ad2c]), partition=
I20240503 02:13:07.122890 32755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24f36508d09446c2ba7bdcec1fcf72ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:07.126452 1484 consensus_queue.cc:1035] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:07.130663 2132 tablet_bootstrap.cc:492] T 24f36508d09446c2ba7bdcec1fcf72ab P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:07.132313 2132 tablet_bootstrap.cc:654] T 24f36508d09446c2ba7bdcec1fcf72ab P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:07.136927 2027 consensus_queue.cc:1035] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:07.149871 2124 tablet_bootstrap.cc:492] T 24f36508d09446c2ba7bdcec1fcf72ab P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:07.150795 2132 tablet_bootstrap.cc:492] T 24f36508d09446c2ba7bdcec1fcf72ab P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:07.151098 2132 ts_tablet_manager.cc:1367] T 24f36508d09446c2ba7bdcec1fcf72ab P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.003s
I20240503 02:13:07.152026 2124 tablet_bootstrap.cc:654] T 24f36508d09446c2ba7bdcec1fcf72ab P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:07.152594 2132 raft_consensus.cc:348] T 24f36508d09446c2ba7bdcec1fcf72ab P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:07.152962 2132 raft_consensus.cc:374] T 24f36508d09446c2ba7bdcec1fcf72ab P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:07.153147 2132 raft_consensus.cc:729] T 24f36508d09446c2ba7bdcec1fcf72ab P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:07.153514 2132 consensus_queue.cc:260] T 24f36508d09446c2ba7bdcec1fcf72ab P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:07.155158 2132 ts_tablet_manager.cc:1398] T 24f36508d09446c2ba7bdcec1fcf72ab P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:13:07.161103 2124 tablet_bootstrap.cc:492] T 24f36508d09446c2ba7bdcec1fcf72ab P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:07.161340 2124 ts_tablet_manager.cc:1367] T 24f36508d09446c2ba7bdcec1fcf72ab P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.005s
I20240503 02:13:07.162170 2124 raft_consensus.cc:348] T 24f36508d09446c2ba7bdcec1fcf72ab P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:07.162490 2124 raft_consensus.cc:374] T 24f36508d09446c2ba7bdcec1fcf72ab P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:07.162619 2124 raft_consensus.cc:729] T 24f36508d09446c2ba7bdcec1fcf72ab P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:07.162953 2124 consensus_queue.cc:260] T 24f36508d09446c2ba7bdcec1fcf72ab P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:07.163765 2124 ts_tablet_manager.cc:1398] T 24f36508d09446c2ba7bdcec1fcf72ab P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:13:07.170141 2195 tablet_bootstrap.cc:492] T 24f36508d09446c2ba7bdcec1fcf72ab P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:07.173838 2195 tablet_bootstrap.cc:654] T 24f36508d09446c2ba7bdcec1fcf72ab P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:07.176061 2195 tablet_bootstrap.cc:492] T 24f36508d09446c2ba7bdcec1fcf72ab P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:07.176251 2195 ts_tablet_manager.cc:1367] T 24f36508d09446c2ba7bdcec1fcf72ab P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 02:13:07.177724 2195 raft_consensus.cc:348] T 24f36508d09446c2ba7bdcec1fcf72ab P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:07.178040 2195 raft_consensus.cc:374] T 24f36508d09446c2ba7bdcec1fcf72ab P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:07.178189 2195 raft_consensus.cc:729] T 24f36508d09446c2ba7bdcec1fcf72ab P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:07.178565 2195 consensus_queue.cc:260] T 24f36508d09446c2ba7bdcec1fcf72ab P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:07.183286 2195 ts_tablet_manager.cc:1398] T 24f36508d09446c2ba7bdcec1fcf72ab P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20240503 02:13:07.281282 2136 raft_consensus.cc:482] T 24f36508d09446c2ba7bdcec1fcf72ab P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:07.281540 2136 raft_consensus.cc:504] T 24f36508d09446c2ba7bdcec1fcf72ab P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:07.282416 2136 leader_election.cc:288] T 24f36508d09446c2ba7bdcec1fcf72ab P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:07.283556 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24f36508d09446c2ba7bdcec1fcf72ab" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:13:07.283880 305 raft_consensus.cc:2454] T 24f36508d09446c2ba7bdcec1fcf72ab P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 0.
I20240503 02:13:07.285140 502 leader_election.cc:302] T 24f36508d09446c2ba7bdcec1fcf72ab P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:07.285668 2136 raft_consensus.cc:2781] T 24f36508d09446c2ba7bdcec1fcf72ab P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:07.285815 2136 raft_consensus.cc:482] T 24f36508d09446c2ba7bdcec1fcf72ab P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:07.285740 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24f36508d09446c2ba7bdcec1fcf72ab" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:13:07.285915 2136 raft_consensus.cc:3037] T 24f36508d09446c2ba7bdcec1fcf72ab P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:07.286015 437 raft_consensus.cc:2454] T 24f36508d09446c2ba7bdcec1fcf72ab P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 0.
I20240503 02:13:07.288890 2136 raft_consensus.cc:504] T 24f36508d09446c2ba7bdcec1fcf72ab P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:07.296348 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24f36508d09446c2ba7bdcec1fcf72ab" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:13:07.296643 437 raft_consensus.cc:3037] T 24f36508d09446c2ba7bdcec1fcf72ab P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:07.297075 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24f36508d09446c2ba7bdcec1fcf72ab" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:13:07.297319 305 raft_consensus.cc:3037] T 24f36508d09446c2ba7bdcec1fcf72ab P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:07.298620 2136 leader_election.cc:288] T 24f36508d09446c2ba7bdcec1fcf72ab P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 election: Requested vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:07.299546 437 raft_consensus.cc:2454] T 24f36508d09446c2ba7bdcec1fcf72ab P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 1.
I20240503 02:13:07.300014 305 raft_consensus.cc:2454] T 24f36508d09446c2ba7bdcec1fcf72ab P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 1.
I20240503 02:13:07.300665 502 leader_election.cc:302] T 24f36508d09446c2ba7bdcec1fcf72ab P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:07.301322 2136 raft_consensus.cc:2781] T 24f36508d09446c2ba7bdcec1fcf72ab P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:07.301486 2136 raft_consensus.cc:686] T 24f36508d09446c2ba7bdcec1fcf72ab P f575ce92c0224eeb80b9fae5f6561fbe [term 1 LEADER]: Becoming Leader. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Running, Role: LEADER
I20240503 02:13:07.301827 2136 consensus_queue.cc:237] T 24f36508d09446c2ba7bdcec1fcf72ab P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:07.311755 29731 catalog_manager.cc:5484] T 24f36508d09446c2ba7bdcec1fcf72ab P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: term changed from 0 to 1, leader changed from <none> to f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8). New cstate: current_term: 1 leader_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:07.338482 29986 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:07.352145 29853 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:07.369752 30245 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:07.386072 30116 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:07.420354 29731 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:13:07.433182 305 raft_consensus.cc:1261] T 24f36508d09446c2ba7bdcec1fcf72ab P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:07.433889 437 raft_consensus.cc:1261] T 24f36508d09446c2ba7bdcec1fcf72ab P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:07.434687 1909 consensus_queue.cc:1035] T 24f36508d09446c2ba7bdcec1fcf72ab P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:07.435163 1909 consensus_queue.cc:1035] T 24f36508d09446c2ba7bdcec1fcf72ab P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:07.438756 1900 consensus_queue.cc:1035] T 9c6e204df01e450598b05c3429574700 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:07.450279 32755 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:13:07.456027 2177 consensus_queue.cc:1035] T 9c6e204df01e450598b05c3429574700 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:07.473414 32620 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:13:07.506114 29731 catalog_manager.cc:5576] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975) does not have the latest schema for tablet 24f36508d09446c2ba7bdcec1fcf72ab (table rebalancer_dynamic_table [id=7026528a8ba24559a989f4be8991ad2c]). Expected version 1 got 0
I20240503 02:13:07.512614 2252 tablet.cc:1722] T 24f36508d09446c2ba7bdcec1fcf72ab P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:07.511449 2251 tablet.cc:1722] T 24f36508d09446c2ba7bdcec1fcf72ab P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:07.513849 2251 tablet.cc:1620] T 24f36508d09446c2ba7bdcec1fcf72ab P f575ce92c0224eeb80b9fae5f6561fbe: MemRowSet was empty: no flush needed.
I20240503 02:13:07.514422 2252 tablet.cc:1620] T 24f36508d09446c2ba7bdcec1fcf72ab P 8a666e798fe8477d8dd1e51b385d5ead: MemRowSet was empty: no flush needed.
I20240503 02:13:07.520150 547 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:07.533455 417 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:07.543923 29730 catalog_manager.cc:5576] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975) does not have the latest schema for tablet 24f36508d09446c2ba7bdcec1fcf72ab (table rebalancer_dynamic_table [id=7026528a8ba24559a989f4be8991ad2c]). Expected version 1 got 0
I20240503 02:13:07.545327 29734 catalog_manager.cc:5576] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241) does not have the latest schema for tablet 24f36508d09446c2ba7bdcec1fcf72ab (table rebalancer_dynamic_table [id=7026528a8ba24559a989f4be8991ad2c]). Expected version 1 got 0
I20240503 02:13:07.545409 2254 tablet.cc:1722] T 24f36508d09446c2ba7bdcec1fcf72ab P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:07.546209 2254 tablet.cc:1620] T 24f36508d09446c2ba7bdcec1fcf72ab P fbbc2be6382e47908ecb7c68c367bd26: MemRowSet was empty: no flush needed.
I20240503 02:13:07.558853 29734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=7026528a8ba24559a989f4be8991ad2c] alter complete (version 1)
I20240503 02:13:07.645541 29731 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_extra_table_22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:07.656755 417 tablet_service.cc:1467] Processing CreateTablet for tablet a6505f95a1c547748fb5cb7a4918b129 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=fedaa5e192b44061a6bd08563b273009]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:07.657289 417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6505f95a1c547748fb5cb7a4918b129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:07.657389 32755 tablet_service.cc:1467] Processing CreateTablet for tablet a6505f95a1c547748fb5cb7a4918b129 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=fedaa5e192b44061a6bd08563b273009]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:07.657982 32755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6505f95a1c547748fb5cb7a4918b129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:07.660694 29986 tablet_service.cc:1467] Processing CreateTablet for tablet a6505f95a1c547748fb5cb7a4918b129 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=fedaa5e192b44061a6bd08563b273009]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:07.661159 29986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6505f95a1c547748fb5cb7a4918b129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:07.661801 547 tablet_service.cc:1467] Processing CreateTablet for tablet 0ec95509de3443bd96bf91a08ef7df4a (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=fedaa5e192b44061a6bd08563b273009]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:07.662214 547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ec95509de3443bd96bf91a08ef7df4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:07.670178 32752 tablet_service.cc:1467] Processing CreateTablet for tablet 51f5f7297e6448d0b0e9afeedad01429 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=fedaa5e192b44061a6bd08563b273009]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:07.670395 29984 tablet_service.cc:1467] Processing CreateTablet for tablet 51f5f7297e6448d0b0e9afeedad01429 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=fedaa5e192b44061a6bd08563b273009]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:07.670377 32620 tablet_service.cc:1467] Processing CreateTablet for tablet 0ec95509de3443bd96bf91a08ef7df4a (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=fedaa5e192b44061a6bd08563b273009]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:07.670634 32752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51f5f7297e6448d0b0e9afeedad01429. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:07.670786 29984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51f5f7297e6448d0b0e9afeedad01429. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:07.670848 32620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ec95509de3443bd96bf91a08ef7df4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:07.673378 32754 tablet_service.cc:1467] Processing CreateTablet for tablet 0ec95509de3443bd96bf91a08ef7df4a (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=fedaa5e192b44061a6bd08563b273009]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:07.673774 32754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ec95509de3443bd96bf91a08ef7df4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:07.677168 29734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:13:07.677590 415 tablet_service.cc:1467] Processing CreateTablet for tablet 51f5f7297e6448d0b0e9afeedad01429 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=fedaa5e192b44061a6bd08563b273009]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:07.678052 415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51f5f7297e6448d0b0e9afeedad01429. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:07.689759 2251 tablet.cc:1722] T 24f36508d09446c2ba7bdcec1fcf72ab P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:07.690299 2251 tablet.cc:1620] T 24f36508d09446c2ba7bdcec1fcf72ab P f575ce92c0224eeb80b9fae5f6561fbe: MemRowSet was empty: no flush needed.
I20240503 02:13:07.691015 2252 tablet.cc:1722] T 24f36508d09446c2ba7bdcec1fcf72ab P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:07.691498 2252 tablet.cc:1620] T 24f36508d09446c2ba7bdcec1fcf72ab P 8a666e798fe8477d8dd1e51b385d5ead: MemRowSet was empty: no flush needed.
I20240503 02:13:07.698731 2254 tablet.cc:1722] T 24f36508d09446c2ba7bdcec1fcf72ab P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:07.699203 2254 tablet.cc:1620] T 24f36508d09446c2ba7bdcec1fcf72ab P fbbc2be6382e47908ecb7c68c367bd26: MemRowSet was empty: no flush needed.
I20240503 02:13:07.703282 29730 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=7026528a8ba24559a989f4be8991ad2c] alter complete (version 2)
I20240503 02:13:07.709759 2273 tablet_bootstrap.cc:492] T 51f5f7297e6448d0b0e9afeedad01429 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:13:07.711069 2274 tablet_bootstrap.cc:492] T 0ec95509de3443bd96bf91a08ef7df4a P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:07.712990 2274 tablet_bootstrap.cc:654] T 0ec95509de3443bd96bf91a08ef7df4a P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:07.714540 2271 tablet_bootstrap.cc:492] T a6505f95a1c547748fb5cb7a4918b129 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:07.716091 2272 tablet_bootstrap.cc:492] T 0ec95509de3443bd96bf91a08ef7df4a P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:07.719362 2273 tablet_bootstrap.cc:654] T 51f5f7297e6448d0b0e9afeedad01429 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:07.719702 2270 tablet_bootstrap.cc:492] T a6505f95a1c547748fb5cb7a4918b129 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:07.721671 2272 tablet_bootstrap.cc:654] T 0ec95509de3443bd96bf91a08ef7df4a P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:07.722076 2270 tablet_bootstrap.cc:654] T a6505f95a1c547748fb5cb7a4918b129 P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:07.723280 2271 tablet_bootstrap.cc:654] T a6505f95a1c547748fb5cb7a4918b129 P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:07.723611 2274 tablet_bootstrap.cc:492] T 0ec95509de3443bd96bf91a08ef7df4a P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:07.723891 2274 ts_tablet_manager.cc:1367] T 0ec95509de3443bd96bf91a08ef7df4a P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.009s
I20240503 02:13:07.725018 2274 raft_consensus.cc:348] T 0ec95509de3443bd96bf91a08ef7df4a P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:07.725414 2274 raft_consensus.cc:374] T 0ec95509de3443bd96bf91a08ef7df4a P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:07.725574 2274 raft_consensus.cc:729] T 0ec95509de3443bd96bf91a08ef7df4a P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:07.725950 2274 consensus_queue.cc:260] T 0ec95509de3443bd96bf91a08ef7df4a P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:07.728416 2274 ts_tablet_manager.cc:1398] T 0ec95509de3443bd96bf91a08ef7df4a P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:13:07.730669 2270 tablet_bootstrap.cc:492] T a6505f95a1c547748fb5cb7a4918b129 P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:07.730844 2270 ts_tablet_manager.cc:1367] T a6505f95a1c547748fb5cb7a4918b129 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20240503 02:13:07.731662 2270 raft_consensus.cc:348] T a6505f95a1c547748fb5cb7a4918b129 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:07.731892 2270 raft_consensus.cc:374] T a6505f95a1c547748fb5cb7a4918b129 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:07.731999 2270 raft_consensus.cc:729] T a6505f95a1c547748fb5cb7a4918b129 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:07.732223 2270 consensus_queue.cc:260] T a6505f95a1c547748fb5cb7a4918b129 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:07.733736 2270 ts_tablet_manager.cc:1398] T a6505f95a1c547748fb5cb7a4918b129 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240503 02:13:07.734081 2270 tablet_bootstrap.cc:492] T 0ec95509de3443bd96bf91a08ef7df4a P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:07.735406 2270 tablet_bootstrap.cc:654] T 0ec95509de3443bd96bf91a08ef7df4a P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:07.738788 30135 consensus_queue.cc:237] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:07.739830 2272 tablet_bootstrap.cc:492] T 0ec95509de3443bd96bf91a08ef7df4a P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:07.740001 2271 tablet_bootstrap.cc:492] T a6505f95a1c547748fb5cb7a4918b129 P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:07.740262 2271 ts_tablet_manager.cc:1367] T a6505f95a1c547748fb5cb7a4918b129 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.000s
I20240503 02:13:07.740530 2272 ts_tablet_manager.cc:1367] T 0ec95509de3443bd96bf91a08ef7df4a P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.000s
I20240503 02:13:07.741838 2272 raft_consensus.cc:348] T 0ec95509de3443bd96bf91a08ef7df4a P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:07.742347 30004 raft_consensus.cc:1261] T 88b57085de3144cea3f64922ae5ccf55 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:07.742368 567 raft_consensus.cc:1261] T 88b57085de3144cea3f64922ae5ccf55 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:07.742574 2271 raft_consensus.cc:348] T a6505f95a1c547748fb5cb7a4918b129 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:07.742856 2271 raft_consensus.cc:374] T a6505f95a1c547748fb5cb7a4918b129 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:07.742995 2272 raft_consensus.cc:374] T 0ec95509de3443bd96bf91a08ef7df4a P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:07.743144 2271 raft_consensus.cc:729] T a6505f95a1c547748fb5cb7a4918b129 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:07.743305 2272 raft_consensus.cc:729] T 0ec95509de3443bd96bf91a08ef7df4a P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:07.743408 1857 consensus_queue.cc:1035] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:07.743633 2271 consensus_queue.cc:260] T a6505f95a1c547748fb5cb7a4918b129 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:07.744117 2272 consensus_queue.cc:260] T 0ec95509de3443bd96bf91a08ef7df4a P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:07.745955 2154 consensus_queue.cc:1035] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:07.746338 2270 tablet_bootstrap.cc:492] T 0ec95509de3443bd96bf91a08ef7df4a P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:07.746496 2270 ts_tablet_manager.cc:1367] T 0ec95509de3443bd96bf91a08ef7df4a P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20240503 02:13:07.748625 2272 ts_tablet_manager.cc:1398] T 0ec95509de3443bd96bf91a08ef7df4a P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:13:07.746500 2271 ts_tablet_manager.cc:1398] T a6505f95a1c547748fb5cb7a4918b129 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:13:07.749141 2271 tablet_bootstrap.cc:492] T 51f5f7297e6448d0b0e9afeedad01429 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:07.747251 2270 raft_consensus.cc:348] T 0ec95509de3443bd96bf91a08ef7df4a P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:07.752977 2270 raft_consensus.cc:374] T 0ec95509de3443bd96bf91a08ef7df4a P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:07.753479 2270 raft_consensus.cc:729] T 0ec95509de3443bd96bf91a08ef7df4a P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:07.753880 2270 consensus_queue.cc:260] T 0ec95509de3443bd96bf91a08ef7df4a P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:07.754568 1857 raft_consensus.cc:2932] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:07.755728 2270 ts_tablet_manager.cc:1398] T 0ec95509de3443bd96bf91a08ef7df4a P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 02:13:07.756047 2270 tablet_bootstrap.cc:492] T 51f5f7297e6448d0b0e9afeedad01429 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:07.758227 2270 tablet_bootstrap.cc:654] T 51f5f7297e6448d0b0e9afeedad01429 P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:07.759368 30004 raft_consensus.cc:2932] T 88b57085de3144cea3f64922ae5ccf55 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:07.760089 2271 tablet_bootstrap.cc:654] T 51f5f7297e6448d0b0e9afeedad01429 P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:07.763057 2271 tablet_bootstrap.cc:492] T 51f5f7297e6448d0b0e9afeedad01429 P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:07.763295 2271 ts_tablet_manager.cc:1367] T 51f5f7297e6448d0b0e9afeedad01429 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240503 02:13:07.769552 29734 catalog_manager.cc:5484] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 reported cstate change: config changed from index 4 to 5, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) evicted. New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:07.771351 2273 tablet_bootstrap.cc:492] T 51f5f7297e6448d0b0e9afeedad01429 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:13:07.771559 2273 ts_tablet_manager.cc:1367] T 51f5f7297e6448d0b0e9afeedad01429 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.062s user 0.003s sys 0.004s
I20240503 02:13:07.772656 2273 raft_consensus.cc:348] T 51f5f7297e6448d0b0e9afeedad01429 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:07.772936 2273 raft_consensus.cc:374] T 51f5f7297e6448d0b0e9afeedad01429 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:07.773725 2273 raft_consensus.cc:729] T 51f5f7297e6448d0b0e9afeedad01429 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:13:07.776510 30245 tablet_service.cc:1514] Processing DeleteTablet for tablet 88b57085de3144cea3f64922ae5ccf55 with delete_type TABLET_DATA_TOMBSTONED (TS 1f10f80f13c84e80b03eda7085a3b012 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:49856
I20240503 02:13:07.774049 2273 consensus_queue.cc:260] T 51f5f7297e6448d0b0e9afeedad01429 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:07.782048 2273 ts_tablet_manager.cc:1398] T 51f5f7297e6448d0b0e9afeedad01429 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.010s user 0.002s sys 0.002s
I20240503 02:13:07.782433 2273 tablet_bootstrap.cc:492] T a6505f95a1c547748fb5cb7a4918b129 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:13:07.789943 2273 tablet_bootstrap.cc:654] T a6505f95a1c547748fb5cb7a4918b129 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:07.789973 2287 tablet_replica.cc:330] T 88b57085de3144cea3f64922ae5ccf55 P 1f10f80f13c84e80b03eda7085a3b012: stopping tablet replica
I20240503 02:13:07.790187 2287 raft_consensus.cc:2229] T 88b57085de3144cea3f64922ae5ccf55 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:07.791316 2287 raft_consensus.cc:2258] T 88b57085de3144cea3f64922ae5ccf55 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:07.792137 2271 raft_consensus.cc:348] T 51f5f7297e6448d0b0e9afeedad01429 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:07.792719 2271 raft_consensus.cc:374] T 51f5f7297e6448d0b0e9afeedad01429 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:07.792892 2271 raft_consensus.cc:729] T 51f5f7297e6448d0b0e9afeedad01429 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:07.793247 2271 consensus_queue.cc:260] T 51f5f7297e6448d0b0e9afeedad01429 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:07.793973 2271 ts_tablet_manager.cc:1398] T 51f5f7297e6448d0b0e9afeedad01429 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.031s user 0.002s sys 0.000s
I20240503 02:13:07.803493 2287 ts_tablet_manager.cc:1875] T 88b57085de3144cea3f64922ae5ccf55 P 1f10f80f13c84e80b03eda7085a3b012: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:07.806311 1787 raft_consensus.cc:482] T 0ec95509de3443bd96bf91a08ef7df4a P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:07.806475 1787 raft_consensus.cc:504] T 0ec95509de3443bd96bf91a08ef7df4a P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:07.807209 1787 leader_election.cc:288] T 0ec95509de3443bd96bf91a08ef7df4a P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:07.808547 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ec95509de3443bd96bf91a08ef7df4a" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:13:07.808604 29730 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:13:07.808796 305 raft_consensus.cc:2454] T 0ec95509de3443bd96bf91a08ef7df4a P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 0.
I20240503 02:13:07.809672 32574 leader_election.cc:302] T 0ec95509de3443bd96bf91a08ef7df4a P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, 9a4d92bc25374fb3960fde7d10be309d; no voters:
I20240503 02:13:07.810264 1787 raft_consensus.cc:2781] T 0ec95509de3443bd96bf91a08ef7df4a P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:07.810387 1787 raft_consensus.cc:482] T 0ec95509de3443bd96bf91a08ef7df4a P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:07.810472 1787 raft_consensus.cc:3037] T 0ec95509de3443bd96bf91a08ef7df4a P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:07.810631 2270 tablet_bootstrap.cc:492] T 51f5f7297e6448d0b0e9afeedad01429 P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:07.810775 2270 ts_tablet_manager.cc:1367] T 51f5f7297e6448d0b0e9afeedad01429 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.055s user 0.017s sys 0.024s
I20240503 02:13:07.813362 2270 raft_consensus.cc:348] T 51f5f7297e6448d0b0e9afeedad01429 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:07.813642 2270 raft_consensus.cc:374] T 51f5f7297e6448d0b0e9afeedad01429 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:07.813766 2270 raft_consensus.cc:729] T 51f5f7297e6448d0b0e9afeedad01429 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:07.813886 2287 ts_tablet_manager.cc:1888] T 88b57085de3144cea3f64922ae5ccf55 P 1f10f80f13c84e80b03eda7085a3b012: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:13:07.814062 2287 log.cc:1196] T 88b57085de3144cea3f64922ae5ccf55 P 1f10f80f13c84e80b03eda7085a3b012: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/wals/88b57085de3144cea3f64922ae5ccf55
I20240503 02:13:07.814975 29719 catalog_manager.cc:4830] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): tablet 88b57085de3144cea3f64922ae5ccf55 (table rebalance_test_table_5 [id=99dbec2a2b63467ab34b96ecc8479244]) successfully deleted
I20240503 02:13:07.815294 566 raft_consensus.cc:2932] T 88b57085de3144cea3f64922ae5ccf55 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:07.824388 2251 tablet.cc:1722] T 24f36508d09446c2ba7bdcec1fcf72ab P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:07.825593 2254 tablet.cc:1722] T 24f36508d09446c2ba7bdcec1fcf72ab P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:07.832342 1787 raft_consensus.cc:504] T 0ec95509de3443bd96bf91a08ef7df4a P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:07.831077 2252 tablet.cc:1722] T 24f36508d09446c2ba7bdcec1fcf72ab P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:07.832829 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ec95509de3443bd96bf91a08ef7df4a" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:07.833082 567 raft_consensus.cc:2454] T 0ec95509de3443bd96bf91a08ef7df4a P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 0.
I20240503 02:13:07.836014 566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ec95509de3443bd96bf91a08ef7df4a" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:13:07.836329 566 raft_consensus.cc:3037] T 0ec95509de3443bd96bf91a08ef7df4a P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:07.835922 29731 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table_new_name [id=7026528a8ba24559a989f4be8991ad2c] alter complete (version 3)
I20240503 02:13:07.838620 2270 consensus_queue.cc:260] T 51f5f7297e6448d0b0e9afeedad01429 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:07.839147 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ec95509de3443bd96bf91a08ef7df4a" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:13:07.839321 2270 ts_tablet_manager.cc:1398] T 51f5f7297e6448d0b0e9afeedad01429 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20240503 02:13:07.839339 305 raft_consensus.cc:3037] T 0ec95509de3443bd96bf91a08ef7df4a P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:07.840471 1787 leader_election.cc:288] T 0ec95509de3443bd96bf91a08ef7df4a P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 election: Requested vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:07.841708 566 raft_consensus.cc:2454] T 0ec95509de3443bd96bf91a08ef7df4a P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 1.
I20240503 02:13:07.842149 305 raft_consensus.cc:2454] T 0ec95509de3443bd96bf91a08ef7df4a P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 1.
I20240503 02:13:07.842180 32573 leader_election.cc:302] T 0ec95509de3443bd96bf91a08ef7df4a P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:07.842957 1787 raft_consensus.cc:2781] T 0ec95509de3443bd96bf91a08ef7df4a P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:07.846328 1787 raft_consensus.cc:686] T 0ec95509de3443bd96bf91a08ef7df4a P 9a4d92bc25374fb3960fde7d10be309d [term 1 LEADER]: Becoming Leader. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Running, Role: LEADER
I20240503 02:13:07.847921 1787 consensus_queue.cc:237] T 0ec95509de3443bd96bf91a08ef7df4a P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:07.849342 2273 tablet_bootstrap.cc:492] T a6505f95a1c547748fb5cb7a4918b129 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:13:07.849622 2273 ts_tablet_manager.cc:1367] T a6505f95a1c547748fb5cb7a4918b129 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.067s user 0.006s sys 0.009s
I20240503 02:13:07.851090 2273 raft_consensus.cc:348] T a6505f95a1c547748fb5cb7a4918b129 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:07.852897 2273 raft_consensus.cc:374] T a6505f95a1c547748fb5cb7a4918b129 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:07.853085 2273 raft_consensus.cc:729] T a6505f95a1c547748fb5cb7a4918b129 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:13:07.853526 2273 consensus_queue.cc:260] T a6505f95a1c547748fb5cb7a4918b129 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:07.854457 2273 ts_tablet_manager.cc:1398] T a6505f95a1c547748fb5cb7a4918b129 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:13:07.856408 29730 catalog_manager.cc:5484] T 0ec95509de3443bd96bf91a08ef7df4a P 9a4d92bc25374fb3960fde7d10be309d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5). New cstate: current_term: 1 leader_uuid: "9a4d92bc25374fb3960fde7d10be309d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:07.938671 29730 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:07.938936 29730 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:07.943559 29730 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet 24f36508d09446c2ba7bdcec1fcf72ab
I20240503 02:13:07.945618 547 tablet_service.cc:1514] Processing DeleteTablet for tablet 24f36508d09446c2ba7bdcec1fcf72ab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:07 UTC) from {username='slave'} at 127.0.0.1:39028
I20240503 02:13:07.945736 417 tablet_service.cc:1514] Processing DeleteTablet for tablet 24f36508d09446c2ba7bdcec1fcf72ab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:07 UTC) from {username='slave'} at 127.0.0.1:50814
I20240503 02:13:07.947086 32755 tablet_service.cc:1514] Processing DeleteTablet for tablet 24f36508d09446c2ba7bdcec1fcf72ab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:07 UTC) from {username='slave'} at 127.0.0.1:46360
I20240503 02:13:07.950951 2303 tablet_replica.cc:330] T 24f36508d09446c2ba7bdcec1fcf72ab P f575ce92c0224eeb80b9fae5f6561fbe: stopping tablet replica
I20240503 02:13:07.951187 2303 raft_consensus.cc:2229] T 24f36508d09446c2ba7bdcec1fcf72ab P f575ce92c0224eeb80b9fae5f6561fbe [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:07.951547 2303 raft_consensus.cc:2258] T 24f36508d09446c2ba7bdcec1fcf72ab P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:07.952243 2305 tablet_replica.cc:330] T 24f36508d09446c2ba7bdcec1fcf72ab P 8a666e798fe8477d8dd1e51b385d5ead: stopping tablet replica
I20240503 02:13:07.952515 2305 raft_consensus.cc:2229] T 24f36508d09446c2ba7bdcec1fcf72ab P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:07.952788 2305 raft_consensus.cc:2258] T 24f36508d09446c2ba7bdcec1fcf72ab P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:07.953747 2305 ts_tablet_manager.cc:1875] T 24f36508d09446c2ba7bdcec1fcf72ab P 8a666e798fe8477d8dd1e51b385d5ead: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:07.959654 2305 ts_tablet_manager.cc:1888] T 24f36508d09446c2ba7bdcec1fcf72ab P 8a666e798fe8477d8dd1e51b385d5ead: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:07.959775 2305 log.cc:1196] T 24f36508d09446c2ba7bdcec1fcf72ab P 8a666e798fe8477d8dd1e51b385d5ead: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal/wals/24f36508d09446c2ba7bdcec1fcf72ab
I20240503 02:13:07.960630 2305 ts_tablet_manager.cc:1909] T 24f36508d09446c2ba7bdcec1fcf72ab P 8a666e798fe8477d8dd1e51b385d5ead: Deleting consensus metadata
I20240503 02:13:07.961621 29719 catalog_manager.cc:4830] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): tablet 24f36508d09446c2ba7bdcec1fcf72ab (table rebalancer_dynamic_table_new_name [id=7026528a8ba24559a989f4be8991ad2c]) successfully deleted
I20240503 02:13:07.963902 2303 ts_tablet_manager.cc:1875] T 24f36508d09446c2ba7bdcec1fcf72ab P f575ce92c0224eeb80b9fae5f6561fbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:07.969470 2304 tablet_replica.cc:330] T 24f36508d09446c2ba7bdcec1fcf72ab P fbbc2be6382e47908ecb7c68c367bd26: stopping tablet replica
I20240503 02:13:07.969787 2304 raft_consensus.cc:2229] T 24f36508d09446c2ba7bdcec1fcf72ab P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:07.970098 2304 raft_consensus.cc:2258] T 24f36508d09446c2ba7bdcec1fcf72ab P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:07.970850 2303 ts_tablet_manager.cc:1888] T 24f36508d09446c2ba7bdcec1fcf72ab P f575ce92c0224eeb80b9fae5f6561fbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:07.971057 2303 log.cc:1196] T 24f36508d09446c2ba7bdcec1fcf72ab P f575ce92c0224eeb80b9fae5f6561fbe: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal/wals/24f36508d09446c2ba7bdcec1fcf72ab
I20240503 02:13:07.971060 2304 ts_tablet_manager.cc:1875] T 24f36508d09446c2ba7bdcec1fcf72ab P fbbc2be6382e47908ecb7c68c367bd26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:07.971607 2303 ts_tablet_manager.cc:1909] T 24f36508d09446c2ba7bdcec1fcf72ab P f575ce92c0224eeb80b9fae5f6561fbe: Deleting consensus metadata
I20240503 02:13:07.972729 29718 catalog_manager.cc:4830] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): tablet 24f36508d09446c2ba7bdcec1fcf72ab (table rebalancer_dynamic_table_new_name [id=7026528a8ba24559a989f4be8991ad2c]) successfully deleted
I20240503 02:13:07.977658 2304 ts_tablet_manager.cc:1888] T 24f36508d09446c2ba7bdcec1fcf72ab P fbbc2be6382e47908ecb7c68c367bd26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:07.977813 2304 log.cc:1196] T 24f36508d09446c2ba7bdcec1fcf72ab P fbbc2be6382e47908ecb7c68c367bd26: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal/wals/24f36508d09446c2ba7bdcec1fcf72ab
I20240503 02:13:07.978245 2304 ts_tablet_manager.cc:1909] T 24f36508d09446c2ba7bdcec1fcf72ab P fbbc2be6382e47908ecb7c68c367bd26: Deleting consensus metadata
I20240503 02:13:07.979391 29719 catalog_manager.cc:4830] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): tablet 24f36508d09446c2ba7bdcec1fcf72ab (table rebalancer_dynamic_table_new_name [id=7026528a8ba24559a989f4be8991ad2c]) successfully deleted
I20240503 02:13:07.984436 2171 raft_consensus.cc:482] T 51f5f7297e6448d0b0e9afeedad01429 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:07.984614 2171 raft_consensus.cc:504] T 51f5f7297e6448d0b0e9afeedad01429 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:07.985386 2171 leader_election.cc:288] T 51f5f7297e6448d0b0e9afeedad01429 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:07.986301 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51f5f7297e6448d0b0e9afeedad01429" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:13:07.986564 305 raft_consensus.cc:2454] T 51f5f7297e6448d0b0e9afeedad01429 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:13:07.987417 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51f5f7297e6448d0b0e9afeedad01429" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:13:07.987646 436 raft_consensus.cc:2454] T 51f5f7297e6448d0b0e9afeedad01429 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:13:07.988054 29940 leader_election.cc:302] T 51f5f7297e6448d0b0e9afeedad01429 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 8a666e798fe8477d8dd1e51b385d5ead; no voters:
I20240503 02:13:07.988396 2171 raft_consensus.cc:2781] T 51f5f7297e6448d0b0e9afeedad01429 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:07.988557 2171 raft_consensus.cc:482] T 51f5f7297e6448d0b0e9afeedad01429 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:07.988656 2171 raft_consensus.cc:3037] T 51f5f7297e6448d0b0e9afeedad01429 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:07.992847 2171 raft_consensus.cc:504] T 51f5f7297e6448d0b0e9afeedad01429 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:07.993585 2171 leader_election.cc:288] T 51f5f7297e6448d0b0e9afeedad01429 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Requested vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:07.994400 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51f5f7297e6448d0b0e9afeedad01429" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:13:07.994475 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51f5f7297e6448d0b0e9afeedad01429" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:13:07.994626 305 raft_consensus.cc:3037] T 51f5f7297e6448d0b0e9afeedad01429 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:07.994663 436 raft_consensus.cc:3037] T 51f5f7297e6448d0b0e9afeedad01429 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:07.997491 436 raft_consensus.cc:2454] T 51f5f7297e6448d0b0e9afeedad01429 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:13:07.997498 305 raft_consensus.cc:2454] T 51f5f7297e6448d0b0e9afeedad01429 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:13:07.997875 29940 leader_election.cc:302] T 51f5f7297e6448d0b0e9afeedad01429 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 8a666e798fe8477d8dd1e51b385d5ead; no voters:
I20240503 02:13:07.998900 2171 raft_consensus.cc:2781] T 51f5f7297e6448d0b0e9afeedad01429 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:07.999114 2171 raft_consensus.cc:686] T 51f5f7297e6448d0b0e9afeedad01429 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Becoming Leader. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Running, Role: LEADER
I20240503 02:13:07.999398 2171 consensus_queue.cc:237] T 51f5f7297e6448d0b0e9afeedad01429 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:08.004580 29730 catalog_manager.cc:5484] T 51f5f7297e6448d0b0e9afeedad01429 P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: term changed from 0 to 1, leader changed from <none> to 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2). New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:08.037436 2171 raft_consensus.cc:482] T a6505f95a1c547748fb5cb7a4918b129 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:08.037654 2171 raft_consensus.cc:504] T a6505f95a1c547748fb5cb7a4918b129 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:08.038309 2171 leader_election.cc:288] T a6505f95a1c547748fb5cb7a4918b129 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:08.038784 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6505f95a1c547748fb5cb7a4918b129" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:13:08.039095 436 raft_consensus.cc:2454] T a6505f95a1c547748fb5cb7a4918b129 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:13:08.039554 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6505f95a1c547748fb5cb7a4918b129" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:13:08.039832 305 raft_consensus.cc:2454] T a6505f95a1c547748fb5cb7a4918b129 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:13:08.040095 29940 leader_election.cc:302] T a6505f95a1c547748fb5cb7a4918b129 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:08.040405 2171 raft_consensus.cc:2781] T a6505f95a1c547748fb5cb7a4918b129 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:08.040524 2171 raft_consensus.cc:482] T a6505f95a1c547748fb5cb7a4918b129 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:08.040640 2171 raft_consensus.cc:3037] T a6505f95a1c547748fb5cb7a4918b129 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:08.045598 2171 raft_consensus.cc:504] T a6505f95a1c547748fb5cb7a4918b129 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:08.046396 2171 leader_election.cc:288] T a6505f95a1c547748fb5cb7a4918b129 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Requested vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:08.047201 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6505f95a1c547748fb5cb7a4918b129" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:13:08.047475 305 raft_consensus.cc:3037] T a6505f95a1c547748fb5cb7a4918b129 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:08.047657 29734 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:13:08.050561 305 raft_consensus.cc:2454] T a6505f95a1c547748fb5cb7a4918b129 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:13:08.051015 29940 leader_election.cc:302] T a6505f95a1c547748fb5cb7a4918b129 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 8a666e798fe8477d8dd1e51b385d5ead; no voters:
I20240503 02:13:08.051054 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6505f95a1c547748fb5cb7a4918b129" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:13:08.051306 2171 raft_consensus.cc:2781] T a6505f95a1c547748fb5cb7a4918b129 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:08.051364 436 raft_consensus.cc:3037] T a6505f95a1c547748fb5cb7a4918b129 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:08.051637 2171 raft_consensus.cc:686] T a6505f95a1c547748fb5cb7a4918b129 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Becoming Leader. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Running, Role: LEADER
I20240503 02:13:08.052004 2171 consensus_queue.cc:237] T a6505f95a1c547748fb5cb7a4918b129 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:08.055073 436 raft_consensus.cc:2454] T a6505f95a1c547748fb5cb7a4918b129 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:13:08.064952 547 tablet_service.cc:1467] Processing CreateTablet for tablet aa0350e091d5409baba8b9b8eefe3373 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2d91404879ea45978c5d3c18259d83b0]), partition=
I20240503 02:13:08.065533 547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa0350e091d5409baba8b9b8eefe3373. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:08.082751 32755 tablet_service.cc:1467] Processing CreateTablet for tablet aa0350e091d5409baba8b9b8eefe3373 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2d91404879ea45978c5d3c18259d83b0]), partition=
I20240503 02:13:08.083189 32755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa0350e091d5409baba8b9b8eefe3373. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:08.088627 2274 tablet_bootstrap.cc:492] T aa0350e091d5409baba8b9b8eefe3373 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:08.089424 417 tablet_service.cc:1467] Processing CreateTablet for tablet aa0350e091d5409baba8b9b8eefe3373 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2d91404879ea45978c5d3c18259d83b0]), partition=
I20240503 02:13:08.089744 417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa0350e091d5409baba8b9b8eefe3373. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:08.090898 2274 tablet_bootstrap.cc:654] T aa0350e091d5409baba8b9b8eefe3373 P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:08.095976 2271 tablet_bootstrap.cc:492] T aa0350e091d5409baba8b9b8eefe3373 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:08.096417 2270 tablet_bootstrap.cc:492] T aa0350e091d5409baba8b9b8eefe3373 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:08.098032 2270 tablet_bootstrap.cc:654] T aa0350e091d5409baba8b9b8eefe3373 P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:08.098062 2271 tablet_bootstrap.cc:654] T aa0350e091d5409baba8b9b8eefe3373 P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:08.104794 2270 tablet_bootstrap.cc:492] T aa0350e091d5409baba8b9b8eefe3373 P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:08.104991 2270 ts_tablet_manager.cc:1367] T aa0350e091d5409baba8b9b8eefe3373 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20240503 02:13:08.105798 2270 raft_consensus.cc:348] T aa0350e091d5409baba8b9b8eefe3373 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:08.106040 2270 raft_consensus.cc:374] T aa0350e091d5409baba8b9b8eefe3373 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:08.106122 2270 raft_consensus.cc:729] T aa0350e091d5409baba8b9b8eefe3373 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:08.106366 2270 consensus_queue.cc:260] T aa0350e091d5409baba8b9b8eefe3373 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:08.106937 2270 ts_tablet_manager.cc:1398] T aa0350e091d5409baba8b9b8eefe3373 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:13:08.113622 29729 catalog_manager.cc:5484] T a6505f95a1c547748fb5cb7a4918b129 P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: term changed from 0 to 1, leader changed from <none> to 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2). New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:08.115284 2271 tablet_bootstrap.cc:492] T aa0350e091d5409baba8b9b8eefe3373 P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:08.115465 2271 ts_tablet_manager.cc:1367] T aa0350e091d5409baba8b9b8eefe3373 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.003s
I20240503 02:13:08.116428 2271 raft_consensus.cc:348] T aa0350e091d5409baba8b9b8eefe3373 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:08.116680 2271 raft_consensus.cc:374] T aa0350e091d5409baba8b9b8eefe3373 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:08.116818 2271 raft_consensus.cc:729] T aa0350e091d5409baba8b9b8eefe3373 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:08.117130 2271 consensus_queue.cc:260] T aa0350e091d5409baba8b9b8eefe3373 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:08.117569 2274 tablet_bootstrap.cc:492] T aa0350e091d5409baba8b9b8eefe3373 P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:08.117796 2274 ts_tablet_manager.cc:1367] T aa0350e091d5409baba8b9b8eefe3373 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.000s
I20240503 02:13:08.117798 2271 ts_tablet_manager.cc:1398] T aa0350e091d5409baba8b9b8eefe3373 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:08.118633 2274 raft_consensus.cc:348] T aa0350e091d5409baba8b9b8eefe3373 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:08.118925 2274 raft_consensus.cc:374] T aa0350e091d5409baba8b9b8eefe3373 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:08.119305 2274 raft_consensus.cc:729] T aa0350e091d5409baba8b9b8eefe3373 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:08.123082 2274 consensus_queue.cc:260] T aa0350e091d5409baba8b9b8eefe3373 P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:08.131214 2274 ts_tablet_manager.cc:1398] T aa0350e091d5409baba8b9b8eefe3373 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240503 02:13:08.204252 29853 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:08.228142 2177 raft_consensus.cc:482] T aa0350e091d5409baba8b9b8eefe3373 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:08.228389 2177 raft_consensus.cc:504] T aa0350e091d5409baba8b9b8eefe3373 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:08.228617 30245 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:08.231114 32755 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:13:08.233377 2177 leader_election.cc:288] T aa0350e091d5409baba8b9b8eefe3373 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:08.234010 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa0350e091d5409baba8b9b8eefe3373" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:08.234015 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa0350e091d5409baba8b9b8eefe3373" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:13:08.234244 567 raft_consensus.cc:2454] T aa0350e091d5409baba8b9b8eefe3373 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 0.
I20240503 02:13:08.234236 307 raft_consensus.cc:2454] T aa0350e091d5409baba8b9b8eefe3373 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 0.
I20240503 02:13:08.234678 370 leader_election.cc:302] T aa0350e091d5409baba8b9b8eefe3373 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f575ce92c0224eeb80b9fae5f6561fbe, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:08.235548 2177 raft_consensus.cc:2781] T aa0350e091d5409baba8b9b8eefe3373 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:08.235667 2177 raft_consensus.cc:482] T aa0350e091d5409baba8b9b8eefe3373 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:08.235786 2177 raft_consensus.cc:3037] T aa0350e091d5409baba8b9b8eefe3373 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:08.241439 547 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:13:08.250607 2177 raft_consensus.cc:504] T aa0350e091d5409baba8b9b8eefe3373 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:08.253139 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa0350e091d5409baba8b9b8eefe3373" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:13:08.253440 567 raft_consensus.cc:3037] T aa0350e091d5409baba8b9b8eefe3373 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:08.258103 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa0350e091d5409baba8b9b8eefe3373" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:13:08.258337 307 raft_consensus.cc:3037] T aa0350e091d5409baba8b9b8eefe3373 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:08.260974 30116 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:08.263126 567 raft_consensus.cc:2454] T aa0350e091d5409baba8b9b8eefe3373 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 1.
I20240503 02:13:08.263320 2177 leader_election.cc:288] T aa0350e091d5409baba8b9b8eefe3373 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Requested vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:08.263967 370 leader_election.cc:302] T aa0350e091d5409baba8b9b8eefe3373 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f575ce92c0224eeb80b9fae5f6561fbe, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:08.264369 2177 raft_consensus.cc:2781] T aa0350e091d5409baba8b9b8eefe3373 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:08.267206 2177 raft_consensus.cc:686] T aa0350e091d5409baba8b9b8eefe3373 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Becoming Leader. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Running, Role: LEADER
I20240503 02:13:08.267558 2177 consensus_queue.cc:237] T aa0350e091d5409baba8b9b8eefe3373 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:08.272639 307 raft_consensus.cc:2454] T aa0350e091d5409baba8b9b8eefe3373 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 1.
I20240503 02:13:08.278275 29734 catalog_manager.cc:5484] T aa0350e091d5409baba8b9b8eefe3373 P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7). New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:08.296454 29984 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:13:08.305260 32620 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:13:08.319661 417 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:08.358476 1787 consensus_queue.cc:1035] T 0ec95509de3443bd96bf91a08ef7df4a P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:08.399291 29734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:13:08.409184 2331 consensus_queue.cc:1035] T 51f5f7297e6448d0b0e9afeedad01429 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:08.425335 307 raft_consensus.cc:1261] T aa0350e091d5409baba8b9b8eefe3373 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:08.426124 1900 consensus_queue.cc:1035] T aa0350e091d5409baba8b9b8eefe3373 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:08.452643 567 raft_consensus.cc:1261] T aa0350e091d5409baba8b9b8eefe3373 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:08.453300 1900 consensus_queue.cc:1035] T aa0350e091d5409baba8b9b8eefe3373 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:08.458663 29734 catalog_manager.cc:5576] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233) does not have the latest schema for tablet aa0350e091d5409baba8b9b8eefe3373 (table rebalancer_dynamic_table [id=2d91404879ea45978c5d3c18259d83b0]). Expected version 1 got 0
I20240503 02:13:08.479540 29734 catalog_manager.cc:5576] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233) does not have the latest schema for tablet aa0350e091d5409baba8b9b8eefe3373 (table rebalancer_dynamic_table [id=2d91404879ea45978c5d3c18259d83b0]). Expected version 1 got 0
W20240503 02:13:08.480669 32574 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.4:33935, user_credentials={real_user=slave}} blocked reactor thread for 46448.8us
I20240503 02:13:08.483006 2171 consensus_queue.cc:1035] T 51f5f7297e6448d0b0e9afeedad01429 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:08.483991 1787 consensus_queue.cc:1035] T 0ec95509de3443bd96bf91a08ef7df4a P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:08.484310 2371 tablet.cc:1722] T aa0350e091d5409baba8b9b8eefe3373 P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:08.484939 2371 tablet.cc:1620] T aa0350e091d5409baba8b9b8eefe3373 P fbbc2be6382e47908ecb7c68c367bd26: MemRowSet was empty: no flush needed.
I20240503 02:13:08.486985 29734 catalog_manager.cc:5576] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975) does not have the latest schema for tablet aa0350e091d5409baba8b9b8eefe3373 (table rebalancer_dynamic_table [id=2d91404879ea45978c5d3c18259d83b0]). Expected version 1 got 0
I20240503 02:13:08.491716 2377 tablet.cc:1722] T aa0350e091d5409baba8b9b8eefe3373 P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:08.492702 2377 tablet.cc:1620] T aa0350e091d5409baba8b9b8eefe3373 P 8a666e798fe8477d8dd1e51b385d5ead: MemRowSet was empty: no flush needed.
I20240503 02:13:08.497354 2372 tablet.cc:1722] T aa0350e091d5409baba8b9b8eefe3373 P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:08.497879 2372 tablet.cc:1620] T aa0350e091d5409baba8b9b8eefe3373 P f575ce92c0224eeb80b9fae5f6561fbe: MemRowSet was empty: no flush needed.
I20240503 02:13:08.510499 29730 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=2d91404879ea45978c5d3c18259d83b0] alter complete (version 1)
I20240503 02:13:08.514003 2171 consensus_queue.cc:1035] T a6505f95a1c547748fb5cb7a4918b129 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:08.523334 2020 consensus_queue.cc:1035] T a6505f95a1c547748fb5cb7a4918b129 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:08.553857 30266 consensus_queue.cc:237] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:08.555972 32641 raft_consensus.cc:1261] T 6bf84757d1fb44689da830a82e857787 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:08.556123 30134 raft_consensus.cc:1261] T 6bf84757d1fb44689da830a82e857787 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:08.556521 2324 consensus_queue.cc:1035] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:08.556978 2324 consensus_queue.cc:1035] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:08.559135 2324 raft_consensus.cc:2932] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:08.559754 32641 raft_consensus.cc:2932] T 6bf84757d1fb44689da830a82e857787 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:08.565223 29734 catalog_manager.cc:5484] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: config changed from index 4 to 5, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) evicted. New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:08.568236 30134 raft_consensus.cc:2932] T 6bf84757d1fb44689da830a82e857787 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:08.570133 29984 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bf84757d1fb44689da830a82e857787 with delete_type TABLET_DATA_TOMBSTONED (TS 155735c7f8b14997ac4c770aee2a9401 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:52158
I20240503 02:13:08.581723 2403 tablet_replica.cc:330] T 6bf84757d1fb44689da830a82e857787 P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:13:08.582059 2403 raft_consensus.cc:2229] T 6bf84757d1fb44689da830a82e857787 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:08.582532 2403 raft_consensus.cc:2258] T 6bf84757d1fb44689da830a82e857787 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:08.583277 2403 ts_tablet_manager.cc:1875] T 6bf84757d1fb44689da830a82e857787 P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:08.589421 2403 ts_tablet_manager.cc:1888] T 6bf84757d1fb44689da830a82e857787 P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:13:08.589661 2403 log.cc:1196] T 6bf84757d1fb44689da830a82e857787 P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/6bf84757d1fb44689da830a82e857787
I20240503 02:13:08.590915 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet 6bf84757d1fb44689da830a82e857787 (table rebalance_test_table_13 [id=df07c1fd4e654fea9a59ea9a0ea79595]) successfully deleted
I20240503 02:13:08.622967 29734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:13:08.631143 2371 tablet.cc:1722] T aa0350e091d5409baba8b9b8eefe3373 P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:08.631717 2371 tablet.cc:1620] T aa0350e091d5409baba8b9b8eefe3373 P fbbc2be6382e47908ecb7c68c367bd26: MemRowSet was empty: no flush needed.
I20240503 02:13:08.631873 2372 tablet.cc:1722] T aa0350e091d5409baba8b9b8eefe3373 P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:08.632328 2372 tablet.cc:1620] T aa0350e091d5409baba8b9b8eefe3373 P f575ce92c0224eeb80b9fae5f6561fbe: MemRowSet was empty: no flush needed.
I20240503 02:13:08.632544 2377 tablet.cc:1722] T aa0350e091d5409baba8b9b8eefe3373 P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:08.633038 2377 tablet.cc:1620] T aa0350e091d5409baba8b9b8eefe3373 P 8a666e798fe8477d8dd1e51b385d5ead: MemRowSet was empty: no flush needed.
I20240503 02:13:08.642190 29734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=2d91404879ea45978c5d3c18259d83b0] alter complete (version 2)
I20240503 02:13:08.675724 29734 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_extra_table_24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:08.690932 32620 tablet_service.cc:1467] Processing CreateTablet for tablet e8a76caac93d441b92e65380bf6dc03e (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=782c508495e74448b6859bfc4ff35de0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:08.691453 32620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8a76caac93d441b92e65380bf6dc03e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:08.691349 417 tablet_service.cc:1467] Processing CreateTablet for tablet e8a76caac93d441b92e65380bf6dc03e (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=782c508495e74448b6859bfc4ff35de0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:08.691730 32755 tablet_service.cc:1467] Processing CreateTablet for tablet eb70e12d4fdb4d15a93d37dfdc659a14 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=782c508495e74448b6859bfc4ff35de0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:08.691841 417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8a76caac93d441b92e65380bf6dc03e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:08.692183 32755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb70e12d4fdb4d15a93d37dfdc659a14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:08.695950 548 tablet_service.cc:1467] Processing CreateTablet for tablet eb70e12d4fdb4d15a93d37dfdc659a14 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=782c508495e74448b6859bfc4ff35de0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:08.695981 415 tablet_service.cc:1467] Processing CreateTablet for tablet eb70e12d4fdb4d15a93d37dfdc659a14 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=782c508495e74448b6859bfc4ff35de0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:08.696739 548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb70e12d4fdb4d15a93d37dfdc659a14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:08.696921 415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb70e12d4fdb4d15a93d37dfdc659a14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:08.697423 547 tablet_service.cc:1467] Processing CreateTablet for tablet e8a76caac93d441b92e65380bf6dc03e (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=782c508495e74448b6859bfc4ff35de0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:08.697854 547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8a76caac93d441b92e65380bf6dc03e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:08.699908 416 tablet_service.cc:1467] Processing CreateTablet for tablet 88f696d03330448d879ed7b60364c28c (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=782c508495e74448b6859bfc4ff35de0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:08.700266 416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88f696d03330448d879ed7b60364c28c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:08.706400 30245 tablet_service.cc:1467] Processing CreateTablet for tablet 88f696d03330448d879ed7b60364c28c (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=782c508495e74448b6859bfc4ff35de0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:08.706921 30245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88f696d03330448d879ed7b60364c28c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:08.708731 2430 tablet_bootstrap.cc:492] T eb70e12d4fdb4d15a93d37dfdc659a14 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:08.710453 2430 tablet_bootstrap.cc:654] T eb70e12d4fdb4d15a93d37dfdc659a14 P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:08.711696 2431 tablet_bootstrap.cc:492] T e8a76caac93d441b92e65380bf6dc03e P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:08.713400 2429 tablet_bootstrap.cc:492] T e8a76caac93d441b92e65380bf6dc03e P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:08.715828 2429 tablet_bootstrap.cc:654] T e8a76caac93d441b92e65380bf6dc03e P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:08.726908 2431 tablet_bootstrap.cc:654] T e8a76caac93d441b92e65380bf6dc03e P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:08.735126 2438 tablet_bootstrap.cc:492] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:08.735698 32620 tablet_service.cc:1467] Processing CreateTablet for tablet 88f696d03330448d879ed7b60364c28c (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=782c508495e74448b6859bfc4ff35de0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:08.736243 32620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88f696d03330448d879ed7b60364c28c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:08.737812 2438 tablet_bootstrap.cc:654] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:08.746567 29734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:13:08.748003 2438 tablet_bootstrap.cc:492] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:08.749323 2438 ts_tablet_manager.cc:1367] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.002s
I20240503 02:13:08.751123 2437 tablet_bootstrap.cc:492] T 88f696d03330448d879ed7b60364c28c P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:13:08.753229 2438 raft_consensus.cc:348] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:08.754518 2438 raft_consensus.cc:374] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:08.754997 2438 raft_consensus.cc:729] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:08.755748 2438 consensus_queue.cc:260] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:08.758433 2431 tablet_bootstrap.cc:492] T e8a76caac93d441b92e65380bf6dc03e P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:08.758603 2431 ts_tablet_manager.cc:1367] T e8a76caac93d441b92e65380bf6dc03e P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.047s user 0.004s sys 0.007s
I20240503 02:13:08.759428 2431 raft_consensus.cc:348] T e8a76caac93d441b92e65380bf6dc03e P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:08.759645 2431 raft_consensus.cc:374] T e8a76caac93d441b92e65380bf6dc03e P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:08.759742 2431 raft_consensus.cc:729] T e8a76caac93d441b92e65380bf6dc03e P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:08.760866 2431 consensus_queue.cc:260] T e8a76caac93d441b92e65380bf6dc03e P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:08.761615 2438 ts_tablet_manager.cc:1398] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 02:13:08.769217 2371 tablet.cc:1722] T aa0350e091d5409baba8b9b8eefe3373 P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:08.775369 2437 tablet_bootstrap.cc:654] T 88f696d03330448d879ed7b60364c28c P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:08.776873 2430 tablet_bootstrap.cc:492] T eb70e12d4fdb4d15a93d37dfdc659a14 P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:08.777122 2430 ts_tablet_manager.cc:1367] T eb70e12d4fdb4d15a93d37dfdc659a14 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.068s user 0.005s sys 0.019s
I20240503 02:13:08.778000 2430 raft_consensus.cc:348] T eb70e12d4fdb4d15a93d37dfdc659a14 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:08.778299 2430 raft_consensus.cc:374] T eb70e12d4fdb4d15a93d37dfdc659a14 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:08.778456 2430 raft_consensus.cc:729] T eb70e12d4fdb4d15a93d37dfdc659a14 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:08.778755 2430 consensus_queue.cc:260] T eb70e12d4fdb4d15a93d37dfdc659a14 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:08.780025 2430 ts_tablet_manager.cc:1398] T eb70e12d4fdb4d15a93d37dfdc659a14 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:08.780345 2430 tablet_bootstrap.cc:492] T e8a76caac93d441b92e65380bf6dc03e P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:08.781500 2372 tablet.cc:1722] T aa0350e091d5409baba8b9b8eefe3373 P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:08.782140 2430 tablet_bootstrap.cc:654] T e8a76caac93d441b92e65380bf6dc03e P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:08.785166 2377 tablet.cc:1722] T aa0350e091d5409baba8b9b8eefe3373 P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:08.792482 2429 tablet_bootstrap.cc:492] T e8a76caac93d441b92e65380bf6dc03e P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:08.792728 2429 ts_tablet_manager.cc:1367] T e8a76caac93d441b92e65380bf6dc03e P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.079s user 0.016s sys 0.012s
I20240503 02:13:08.793079 29730 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table_new_name [id=2d91404879ea45978c5d3c18259d83b0] alter complete (version 3)
I20240503 02:13:08.793773 2429 raft_consensus.cc:348] T e8a76caac93d441b92e65380bf6dc03e P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:08.794028 2429 raft_consensus.cc:374] T e8a76caac93d441b92e65380bf6dc03e P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:08.794178 2429 raft_consensus.cc:729] T e8a76caac93d441b92e65380bf6dc03e P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:08.794602 2429 consensus_queue.cc:260] T e8a76caac93d441b92e65380bf6dc03e P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:08.795246 2429 ts_tablet_manager.cc:1398] T e8a76caac93d441b92e65380bf6dc03e P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:13:08.796396 2429 tablet_bootstrap.cc:492] T 88f696d03330448d879ed7b60364c28c P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:08.798169 2429 tablet_bootstrap.cc:654] T 88f696d03330448d879ed7b60364c28c P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:08.804807 2429 tablet_bootstrap.cc:492] T 88f696d03330448d879ed7b60364c28c P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:08.805446 2429 ts_tablet_manager.cc:1367] T 88f696d03330448d879ed7b60364c28c P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.001s
I20240503 02:13:08.806295 2429 raft_consensus.cc:348] T 88f696d03330448d879ed7b60364c28c P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:08.806619 2429 raft_consensus.cc:374] T 88f696d03330448d879ed7b60364c28c P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:08.806741 2429 raft_consensus.cc:729] T 88f696d03330448d879ed7b60364c28c P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:08.807057 2429 consensus_queue.cc:260] T 88f696d03330448d879ed7b60364c28c P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:08.807734 2429 ts_tablet_manager.cc:1398] T 88f696d03330448d879ed7b60364c28c P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:13:08.815394 1787 raft_consensus.cc:482] T e8a76caac93d441b92e65380bf6dc03e P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:08.815577 1787 raft_consensus.cc:504] T e8a76caac93d441b92e65380bf6dc03e P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:08.816228 1787 leader_election.cc:288] T e8a76caac93d441b92e65380bf6dc03e P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:13:08.816639 2431 ts_tablet_manager.cc:1398] T e8a76caac93d441b92e65380bf6dc03e P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.058s user 0.002s sys 0.000s
I20240503 02:13:08.817951 2431 tablet_bootstrap.cc:492] T eb70e12d4fdb4d15a93d37dfdc659a14 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:08.819464 566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a76caac93d441b92e65380bf6dc03e" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:08.819711 566 raft_consensus.cc:2454] T e8a76caac93d441b92e65380bf6dc03e P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 0.
I20240503 02:13:08.820833 32573 leader_election.cc:302] T e8a76caac93d441b92e65380bf6dc03e P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:08.821213 1787 raft_consensus.cc:2781] T e8a76caac93d441b92e65380bf6dc03e P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:08.821295 1787 raft_consensus.cc:482] T e8a76caac93d441b92e65380bf6dc03e P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:08.821393 1787 raft_consensus.cc:3037] T e8a76caac93d441b92e65380bf6dc03e P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:08.824139 1787 raft_consensus.cc:504] T e8a76caac93d441b92e65380bf6dc03e P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:08.826395 566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a76caac93d441b92e65380bf6dc03e" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:13:08.826637 566 raft_consensus.cc:3037] T e8a76caac93d441b92e65380bf6dc03e P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:08.826813 2431 tablet_bootstrap.cc:654] T eb70e12d4fdb4d15a93d37dfdc659a14 P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:08.830983 2431 tablet_bootstrap.cc:492] T eb70e12d4fdb4d15a93d37dfdc659a14 P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:08.830961 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a76caac93d441b92e65380bf6dc03e" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:13:08.831141 2431 ts_tablet_manager.cc:1367] T eb70e12d4fdb4d15a93d37dfdc659a14 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.002s
I20240503 02:13:08.831395 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a76caac93d441b92e65380bf6dc03e" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:13:08.831959 2431 raft_consensus.cc:348] T eb70e12d4fdb4d15a93d37dfdc659a14 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:08.832180 2431 raft_consensus.cc:374] T eb70e12d4fdb4d15a93d37dfdc659a14 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:08.832252 2431 raft_consensus.cc:729] T eb70e12d4fdb4d15a93d37dfdc659a14 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
W20240503 02:13:08.832178 32574 leader_election.cc:341] T e8a76caac93d441b92e65380bf6dc03e P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:08.832466 2431 consensus_queue.cc:260] T eb70e12d4fdb4d15a93d37dfdc659a14 P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
W20240503 02:13:08.832641 32574 leader_election.cc:341] T e8a76caac93d441b92e65380bf6dc03e P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:08.833014 2431 ts_tablet_manager.cc:1398] T eb70e12d4fdb4d15a93d37dfdc659a14 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:13:08.838702 566 raft_consensus.cc:2454] T e8a76caac93d441b92e65380bf6dc03e P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 1.
I20240503 02:13:08.841164 1787 leader_election.cc:288] T e8a76caac93d441b92e65380bf6dc03e P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 election: Requested vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:13:08.841176 32573 leader_election.cc:302] T e8a76caac93d441b92e65380bf6dc03e P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe; no voters: fbbc2be6382e47908ecb7c68c367bd26
I20240503 02:13:08.842267 2430 tablet_bootstrap.cc:492] T e8a76caac93d441b92e65380bf6dc03e P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:08.842352 2369 raft_consensus.cc:2781] T e8a76caac93d441b92e65380bf6dc03e P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:08.842572 2430 ts_tablet_manager.cc:1367] T e8a76caac93d441b92e65380bf6dc03e P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.062s user 0.007s sys 0.005s
I20240503 02:13:08.842604 2369 raft_consensus.cc:686] T e8a76caac93d441b92e65380bf6dc03e P 9a4d92bc25374fb3960fde7d10be309d [term 1 LEADER]: Becoming Leader. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Running, Role: LEADER
I20240503 02:13:08.842938 2369 consensus_queue.cc:237] T e8a76caac93d441b92e65380bf6dc03e P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:08.843675 2430 raft_consensus.cc:348] T e8a76caac93d441b92e65380bf6dc03e P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:08.843987 2430 raft_consensus.cc:374] T e8a76caac93d441b92e65380bf6dc03e P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:08.844167 2430 raft_consensus.cc:729] T e8a76caac93d441b92e65380bf6dc03e P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:08.844477 2430 consensus_queue.cc:260] T e8a76caac93d441b92e65380bf6dc03e P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:08.845425 2430 ts_tablet_manager.cc:1398] T e8a76caac93d441b92e65380bf6dc03e P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:13:08.845757 2430 tablet_bootstrap.cc:492] T 88f696d03330448d879ed7b60364c28c P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:08.847460 2430 tablet_bootstrap.cc:654] T 88f696d03330448d879ed7b60364c28c P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:08.857586 2430 tablet_bootstrap.cc:492] T 88f696d03330448d879ed7b60364c28c P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:08.857837 2430 ts_tablet_manager.cc:1367] T 88f696d03330448d879ed7b60364c28c P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.004s
I20240503 02:13:08.859063 2430 raft_consensus.cc:348] T 88f696d03330448d879ed7b60364c28c P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:08.859293 2430 raft_consensus.cc:374] T 88f696d03330448d879ed7b60364c28c P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:08.859392 2430 raft_consensus.cc:729] T 88f696d03330448d879ed7b60364c28c P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:08.859640 2430 consensus_queue.cc:260] T 88f696d03330448d879ed7b60364c28c P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:08.860158 2430 ts_tablet_manager.cc:1398] T 88f696d03330448d879ed7b60364c28c P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:13:08.876371 29734 catalog_manager.cc:5484] T e8a76caac93d441b92e65380bf6dc03e P 9a4d92bc25374fb3960fde7d10be309d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5). New cstate: current_term: 1 leader_uuid: "9a4d92bc25374fb3960fde7d10be309d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:08.881848 2375 raft_consensus.cc:482] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:08.882092 2437 tablet_bootstrap.cc:492] T 88f696d03330448d879ed7b60364c28c P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:13:08.882043 2375 raft_consensus.cc:504] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:08.882383 2437 ts_tablet_manager.cc:1367] T 88f696d03330448d879ed7b60364c28c P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.131s user 0.007s sys 0.000s
I20240503 02:13:08.882803 2375 leader_election.cc:288] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:08.883358 2437 raft_consensus.cc:348] T 88f696d03330448d879ed7b60364c28c P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:08.884929 2437 raft_consensus.cc:374] T 88f696d03330448d879ed7b60364c28c P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:08.885111 2437 raft_consensus.cc:729] T 88f696d03330448d879ed7b60364c28c P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:13:08.885535 2437 consensus_queue.cc:260] T 88f696d03330448d879ed7b60364c28c P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:08.886309 2437 ts_tablet_manager.cc:1398] T 88f696d03330448d879ed7b60364c28c P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:13:08.890712 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb70e12d4fdb4d15a93d37dfdc659a14" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:13:08.890957 437 raft_consensus.cc:2454] T eb70e12d4fdb4d15a93d37dfdc659a14 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:13:08.893451 566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb70e12d4fdb4d15a93d37dfdc659a14" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:08.893714 566 raft_consensus.cc:2454] T eb70e12d4fdb4d15a93d37dfdc659a14 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:13:08.896812 32709 leader_election.cc:302] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:08.897143 2375 raft_consensus.cc:2781] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:08.897264 2375 raft_consensus.cc:482] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:08.897344 2375 raft_consensus.cc:3037] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:08.912979 2375 raft_consensus.cc:504] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:08.914508 566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb70e12d4fdb4d15a93d37dfdc659a14" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:13:08.914741 566 raft_consensus.cc:3037] T eb70e12d4fdb4d15a93d37dfdc659a14 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:08.917639 2375 leader_election.cc:288] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Requested vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:08.918571 566 raft_consensus.cc:2454] T eb70e12d4fdb4d15a93d37dfdc659a14 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:13:08.919199 32708 leader_election.cc:302] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:08.919363 29731 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:08.919569 29731 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:08.919930 2375 raft_consensus.cc:2781] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:08.920470 2375 raft_consensus.cc:686] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Becoming Leader. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Running, Role: LEADER
I20240503 02:13:08.921375 2375 consensus_queue.cc:237] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:08.923700 29731 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet aa0350e091d5409baba8b9b8eefe3373
I20240503 02:13:08.925284 32755 tablet_service.cc:1514] Processing DeleteTablet for tablet aa0350e091d5409baba8b9b8eefe3373 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:08 UTC) from {username='slave'} at 127.0.0.1:46360
I20240503 02:13:08.925734 416 tablet_service.cc:1514] Processing DeleteTablet for tablet aa0350e091d5409baba8b9b8eefe3373 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:08 UTC) from {username='slave'} at 127.0.0.1:50814
I20240503 02:13:08.926286 548 tablet_service.cc:1514] Processing DeleteTablet for tablet aa0350e091d5409baba8b9b8eefe3373 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:08 UTC) from {username='slave'} at 127.0.0.1:39028
I20240503 02:13:08.979192 2456 tablet_replica.cc:330] T aa0350e091d5409baba8b9b8eefe3373 P f575ce92c0224eeb80b9fae5f6561fbe: stopping tablet replica
I20240503 02:13:08.979547 2456 raft_consensus.cc:2229] T aa0350e091d5409baba8b9b8eefe3373 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:08.979898 2456 raft_consensus.cc:2258] T aa0350e091d5409baba8b9b8eefe3373 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:08.980453 2455 tablet_replica.cc:330] T aa0350e091d5409baba8b9b8eefe3373 P fbbc2be6382e47908ecb7c68c367bd26: stopping tablet replica
I20240503 02:13:08.980752 2455 raft_consensus.cc:2229] T aa0350e091d5409baba8b9b8eefe3373 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:08.981199 2455 raft_consensus.cc:2258] T aa0350e091d5409baba8b9b8eefe3373 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:08.982009 2454 tablet_replica.cc:330] T aa0350e091d5409baba8b9b8eefe3373 P 8a666e798fe8477d8dd1e51b385d5ead: stopping tablet replica
I20240503 02:13:08.982273 2454 raft_consensus.cc:2229] T aa0350e091d5409baba8b9b8eefe3373 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:08.982544 2454 raft_consensus.cc:2258] T aa0350e091d5409baba8b9b8eefe3373 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:08.985658 2455 ts_tablet_manager.cc:1875] T aa0350e091d5409baba8b9b8eefe3373 P fbbc2be6382e47908ecb7c68c367bd26: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:13:08.986212 32709 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.5:35595, user_credentials={real_user=slave}} blocked reactor thread for 73788.3us
I20240503 02:13:08.987138 2456 ts_tablet_manager.cc:1875] T aa0350e091d5409baba8b9b8eefe3373 P f575ce92c0224eeb80b9fae5f6561fbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:08.987882 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb70e12d4fdb4d15a93d37dfdc659a14" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:13:08.988090 437 raft_consensus.cc:3037] T eb70e12d4fdb4d15a93d37dfdc659a14 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:08.990999 437 raft_consensus.cc:2454] T eb70e12d4fdb4d15a93d37dfdc659a14 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:13:08.986881 2454 ts_tablet_manager.cc:1875] T aa0350e091d5409baba8b9b8eefe3373 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:09.003257 2456 ts_tablet_manager.cc:1888] T aa0350e091d5409baba8b9b8eefe3373 P f575ce92c0224eeb80b9fae5f6561fbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:09.004688 29731 catalog_manager.cc:5484] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6). New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:09.006906 2455 ts_tablet_manager.cc:1888] T aa0350e091d5409baba8b9b8eefe3373 P fbbc2be6382e47908ecb7c68c367bd26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:09.010814 2455 log.cc:1196] T aa0350e091d5409baba8b9b8eefe3373 P fbbc2be6382e47908ecb7c68c367bd26: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal/wals/aa0350e091d5409baba8b9b8eefe3373
I20240503 02:13:09.011543 2456 log.cc:1196] T aa0350e091d5409baba8b9b8eefe3373 P f575ce92c0224eeb80b9fae5f6561fbe: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal/wals/aa0350e091d5409baba8b9b8eefe3373
I20240503 02:13:09.011811 2455 ts_tablet_manager.cc:1909] T aa0350e091d5409baba8b9b8eefe3373 P fbbc2be6382e47908ecb7c68c367bd26: Deleting consensus metadata
I20240503 02:13:09.013556 29719 catalog_manager.cc:4830] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): tablet aa0350e091d5409baba8b9b8eefe3373 (table rebalancer_dynamic_table_new_name [id=2d91404879ea45978c5d3c18259d83b0]) successfully deleted
I20240503 02:13:09.013804 2456 ts_tablet_manager.cc:1909] T aa0350e091d5409baba8b9b8eefe3373 P f575ce92c0224eeb80b9fae5f6561fbe: Deleting consensus metadata
I20240503 02:13:09.018010 2454 ts_tablet_manager.cc:1888] T aa0350e091d5409baba8b9b8eefe3373 P 8a666e798fe8477d8dd1e51b385d5ead: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:09.018344 2454 log.cc:1196] T aa0350e091d5409baba8b9b8eefe3373 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal/wals/aa0350e091d5409baba8b9b8eefe3373
I20240503 02:13:09.018985 2454 ts_tablet_manager.cc:1909] T aa0350e091d5409baba8b9b8eefe3373 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting consensus metadata
I20240503 02:13:09.019806 29718 catalog_manager.cc:4830] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): tablet aa0350e091d5409baba8b9b8eefe3373 (table rebalancer_dynamic_table_new_name [id=2d91404879ea45978c5d3c18259d83b0]) successfully deleted
I20240503 02:13:09.021049 29719 catalog_manager.cc:4830] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): tablet aa0350e091d5409baba8b9b8eefe3373 (table rebalancer_dynamic_table_new_name [id=2d91404879ea45978c5d3c18259d83b0]) successfully deleted
I20240503 02:13:09.026811 29731 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:13:09.037446 548 tablet_service.cc:1467] Processing CreateTablet for tablet dd90eef09b7343a09cfda4554e67d872 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b7ee08c198394bb4a152be45d2d9199c]), partition=
I20240503 02:13:09.037479 416 tablet_service.cc:1467] Processing CreateTablet for tablet dd90eef09b7343a09cfda4554e67d872 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b7ee08c198394bb4a152be45d2d9199c]), partition=
I20240503 02:13:09.037838 548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd90eef09b7343a09cfda4554e67d872. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:09.037896 416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd90eef09b7343a09cfda4554e67d872. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:09.040804 29853 tablet_service.cc:1467] Processing CreateTablet for tablet dd90eef09b7343a09cfda4554e67d872 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b7ee08c198394bb4a152be45d2d9199c]), partition=
I20240503 02:13:09.041196 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd90eef09b7343a09cfda4554e67d872. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:09.044083 2430 tablet_bootstrap.cc:492] T dd90eef09b7343a09cfda4554e67d872 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:09.045586 2430 tablet_bootstrap.cc:654] T dd90eef09b7343a09cfda4554e67d872 P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:09.059691 2430 tablet_bootstrap.cc:492] T dd90eef09b7343a09cfda4554e67d872 P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:09.059988 2430 ts_tablet_manager.cc:1367] T dd90eef09b7343a09cfda4554e67d872 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.002s
I20240503 02:13:09.060850 2430 raft_consensus.cc:348] T dd90eef09b7343a09cfda4554e67d872 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:09.061251 2430 raft_consensus.cc:374] T dd90eef09b7343a09cfda4554e67d872 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:09.061386 2430 raft_consensus.cc:729] T dd90eef09b7343a09cfda4554e67d872 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:09.061712 2430 consensus_queue.cc:260] T dd90eef09b7343a09cfda4554e67d872 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:09.062443 2430 ts_tablet_manager.cc:1398] T dd90eef09b7343a09cfda4554e67d872 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:09.067371 2431 tablet_bootstrap.cc:492] T dd90eef09b7343a09cfda4554e67d872 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:09.069111 2431 tablet_bootstrap.cc:654] T dd90eef09b7343a09cfda4554e67d872 P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:09.075443 2431 tablet_bootstrap.cc:492] T dd90eef09b7343a09cfda4554e67d872 P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:09.075616 2431 ts_tablet_manager.cc:1367] T dd90eef09b7343a09cfda4554e67d872 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20240503 02:13:09.075807 2464 tablet_bootstrap.cc:492] T dd90eef09b7343a09cfda4554e67d872 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:13:09.076691 2431 raft_consensus.cc:348] T dd90eef09b7343a09cfda4554e67d872 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:09.076963 2431 raft_consensus.cc:374] T dd90eef09b7343a09cfda4554e67d872 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:09.077083 2431 raft_consensus.cc:729] T dd90eef09b7343a09cfda4554e67d872 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:09.077347 2431 consensus_queue.cc:260] T dd90eef09b7343a09cfda4554e67d872 P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:09.078383 2464 tablet_bootstrap.cc:654] T dd90eef09b7343a09cfda4554e67d872 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:09.078758 2431 ts_tablet_manager.cc:1398] T dd90eef09b7343a09cfda4554e67d872 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240503 02:13:09.098570 2464 tablet_bootstrap.cc:492] T dd90eef09b7343a09cfda4554e67d872 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:13:09.098759 2464 ts_tablet_manager.cc:1367] T dd90eef09b7343a09cfda4554e67d872 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20240503 02:13:09.099766 2464 raft_consensus.cc:348] T dd90eef09b7343a09cfda4554e67d872 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:09.100054 2464 raft_consensus.cc:374] T dd90eef09b7343a09cfda4554e67d872 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:09.100133 2464 raft_consensus.cc:729] T dd90eef09b7343a09cfda4554e67d872 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:13:09.100328 2464 consensus_queue.cc:260] T dd90eef09b7343a09cfda4554e67d872 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:09.100937 2464 ts_tablet_manager.cc:1398] T dd90eef09b7343a09cfda4554e67d872 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:13:09.104135 29984 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:13:09.111217 1900 raft_consensus.cc:482] T 88f696d03330448d879ed7b60364c28c P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:09.111456 1900 raft_consensus.cc:504] T 88f696d03330448d879ed7b60364c28c P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:09.112269 1900 leader_election.cc:288] T 88f696d03330448d879ed7b60364c28c P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:13:09.113420 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f696d03330448d879ed7b60364c28c" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:13:09.113667 32640 raft_consensus.cc:2454] T 88f696d03330448d879ed7b60364c28c P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 0.
I20240503 02:13:09.114274 371 leader_election.cc:302] T 88f696d03330448d879ed7b60364c28c P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:09.114668 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f696d03330448d879ed7b60364c28c" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:13:09.114943 30266 raft_consensus.cc:2454] T 88f696d03330448d879ed7b60364c28c P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 0.
I20240503 02:13:09.115586 1900 raft_consensus.cc:2781] T 88f696d03330448d879ed7b60364c28c P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:09.115685 1900 raft_consensus.cc:482] T 88f696d03330448d879ed7b60364c28c P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:09.115796 1900 raft_consensus.cc:3037] T 88f696d03330448d879ed7b60364c28c P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:09.119127 1900 raft_consensus.cc:504] T 88f696d03330448d879ed7b60364c28c P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:09.119789 1900 leader_election.cc:288] T 88f696d03330448d879ed7b60364c28c P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Requested vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:13:09.120815 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f696d03330448d879ed7b60364c28c" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d"
I20240503 02:13:09.121119 32641 raft_consensus.cc:3037] T 88f696d03330448d879ed7b60364c28c P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:09.121227 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f696d03330448d879ed7b60364c28c" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:13:09.121428 30266 raft_consensus.cc:3037] T 88f696d03330448d879ed7b60364c28c P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:09.126751 32641 raft_consensus.cc:2454] T 88f696d03330448d879ed7b60364c28c P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 1.
I20240503 02:13:09.129173 371 leader_election.cc:302] T 88f696d03330448d879ed7b60364c28c P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:09.129693 2412 raft_consensus.cc:2781] T 88f696d03330448d879ed7b60364c28c P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:09.129894 2412 raft_consensus.cc:686] T 88f696d03330448d879ed7b60364c28c P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Becoming Leader. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Running, Role: LEADER
I20240503 02:13:09.130297 2412 consensus_queue.cc:237] T 88f696d03330448d879ed7b60364c28c P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:09.132639 30266 raft_consensus.cc:2454] T 88f696d03330448d879ed7b60364c28c P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 1.
I20240503 02:13:09.146212 29731 catalog_manager.cc:5484] T 88f696d03330448d879ed7b60364c28c P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7). New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:09.168474 30245 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:09.185564 29853 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:09.215970 32620 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:09.225476 2412 raft_consensus.cc:482] T e8a76caac93d441b92e65380bf6dc03e P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:09.225718 2412 raft_consensus.cc:504] T e8a76caac93d441b92e65380bf6dc03e P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:09.226051 437 raft_consensus.cc:3037] T e8a76caac93d441b92e65380bf6dc03e P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:09.226513 2412 leader_election.cc:288] T e8a76caac93d441b92e65380bf6dc03e P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:13:09.230953 566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a76caac93d441b92e65380bf6dc03e" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:09.231375 566 raft_consensus.cc:2379] T e8a76caac93d441b92e65380bf6dc03e P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fbbc2be6382e47908ecb7c68c367bd26 in current term 1: Already voted for candidate 9a4d92bc25374fb3960fde7d10be309d in this term.
I20240503 02:13:09.232312 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a76caac93d441b92e65380bf6dc03e" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:13:09.232930 2369 consensus_queue.cc:1035] T e8a76caac93d441b92e65380bf6dc03e P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:09.233111 371 leader_election.cc:302] T e8a76caac93d441b92e65380bf6dc03e P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:13:09.233876 2412 raft_consensus.cc:2726] T e8a76caac93d441b92e65380bf6dc03e P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:09.235100 2369 consensus_queue.cc:1035] T e8a76caac93d441b92e65380bf6dc03e P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:09.251585 32755 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:09.285629 548 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:13:09.310248 30116 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:09.321569 416 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:09.388490 2375 consensus_queue.cc:1035] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:09.411177 2375 consensus_queue.cc:1035] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:09.425313 1909 raft_consensus.cc:482] T dd90eef09b7343a09cfda4554e67d872 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:09.425495 1909 raft_consensus.cc:504] T dd90eef09b7343a09cfda4554e67d872 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:09.426203 1909 leader_election.cc:288] T dd90eef09b7343a09cfda4554e67d872 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:09.428419 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd90eef09b7343a09cfda4554e67d872" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:13:09.428489 29872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd90eef09b7343a09cfda4554e67d872" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:13:09.428675 437 raft_consensus.cc:2454] T dd90eef09b7343a09cfda4554e67d872 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 0.
I20240503 02:13:09.428771 29872 raft_consensus.cc:2454] T dd90eef09b7343a09cfda4554e67d872 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 0.
I20240503 02:13:09.429150 502 leader_election.cc:302] T dd90eef09b7343a09cfda4554e67d872 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f575ce92c0224eeb80b9fae5f6561fbe, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:09.429454 1909 raft_consensus.cc:2781] T dd90eef09b7343a09cfda4554e67d872 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:09.429592 1909 raft_consensus.cc:482] T dd90eef09b7343a09cfda4554e67d872 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:09.429677 1909 raft_consensus.cc:3037] T dd90eef09b7343a09cfda4554e67d872 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:09.433293 1909 raft_consensus.cc:504] T dd90eef09b7343a09cfda4554e67d872 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:09.434603 29872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd90eef09b7343a09cfda4554e67d872" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:13:09.434828 29872 raft_consensus.cc:3037] T dd90eef09b7343a09cfda4554e67d872 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:09.434829 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd90eef09b7343a09cfda4554e67d872" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:13:09.435084 437 raft_consensus.cc:3037] T dd90eef09b7343a09cfda4554e67d872 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:09.437687 29872 raft_consensus.cc:2454] T dd90eef09b7343a09cfda4554e67d872 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 1.
I20240503 02:13:09.439517 437 raft_consensus.cc:2454] T dd90eef09b7343a09cfda4554e67d872 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 1.
I20240503 02:13:09.440073 502 leader_election.cc:302] T dd90eef09b7343a09cfda4554e67d872 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f575ce92c0224eeb80b9fae5f6561fbe, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:09.442142 1909 leader_election.cc:288] T dd90eef09b7343a09cfda4554e67d872 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 election: Requested vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:09.443033 1909 raft_consensus.cc:2781] T dd90eef09b7343a09cfda4554e67d872 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:09.444037 1909 raft_consensus.cc:686] T dd90eef09b7343a09cfda4554e67d872 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 LEADER]: Becoming Leader. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Running, Role: LEADER
I20240503 02:13:09.444700 1909 consensus_queue.cc:237] T dd90eef09b7343a09cfda4554e67d872 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:09.461786 29734 catalog_manager.cc:5484] T dd90eef09b7343a09cfda4554e67d872 P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: term changed from 0 to 1, leader changed from <none> to f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8). New cstate: current_term: 1 leader_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:09.506235 435 consensus_queue.cc:237] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:09.508409 32641 raft_consensus.cc:1261] T 83c7d1060ccf49a2adda4302600a9876 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:09.508419 30266 raft_consensus.cc:1261] T 83c7d1060ccf49a2adda4302600a9876 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:09.508958 1900 consensus_queue.cc:1035] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:09.509325 2412 consensus_queue.cc:1035] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:09.511826 1900 raft_consensus.cc:2932] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:09.512454 30266 raft_consensus.cc:2932] T 83c7d1060ccf49a2adda4302600a9876 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:09.512809 32641 raft_consensus.cc:2932] T 83c7d1060ccf49a2adda4302600a9876 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:09.518275 29734 catalog_manager.cc:5484] T 83c7d1060ccf49a2adda4302600a9876 P 9a4d92bc25374fb3960fde7d10be309d reported cstate change: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:09.523483 548 tablet_service.cc:1514] Processing DeleteTablet for tablet 83c7d1060ccf49a2adda4302600a9876 with delete_type TABLET_DATA_TOMBSTONED (TS f575ce92c0224eeb80b9fae5f6561fbe not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:39028
I20240503 02:13:09.524181 2515 tablet_replica.cc:330] T 83c7d1060ccf49a2adda4302600a9876 P f575ce92c0224eeb80b9fae5f6561fbe: stopping tablet replica
I20240503 02:13:09.524411 2515 raft_consensus.cc:2229] T 83c7d1060ccf49a2adda4302600a9876 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:09.524632 2515 raft_consensus.cc:2258] T 83c7d1060ccf49a2adda4302600a9876 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:09.525235 2515 ts_tablet_manager.cc:1875] T 83c7d1060ccf49a2adda4302600a9876 P f575ce92c0224eeb80b9fae5f6561fbe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:09.531353 2515 ts_tablet_manager.cc:1888] T 83c7d1060ccf49a2adda4302600a9876 P f575ce92c0224eeb80b9fae5f6561fbe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:13:09.531476 2515 log.cc:1196] T 83c7d1060ccf49a2adda4302600a9876 P f575ce92c0224eeb80b9fae5f6561fbe: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal/wals/83c7d1060ccf49a2adda4302600a9876
I20240503 02:13:09.532258 29718 catalog_manager.cc:4830] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): tablet 83c7d1060ccf49a2adda4302600a9876 (table rebalancer_extra_table_0 [id=c9cc90e5637140278312ab93695a8afd]) successfully deleted
I20240503 02:13:09.569873 29730 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:13:09.575934 435 raft_consensus.cc:1261] T dd90eef09b7343a09cfda4554e67d872 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:09.575992 29872 raft_consensus.cc:1261] T dd90eef09b7343a09cfda4554e67d872 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:09.576701 2494 consensus_queue.cc:1035] T dd90eef09b7343a09cfda4554e67d872 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:09.577248 2494 consensus_queue.cc:1035] T dd90eef09b7343a09cfda4554e67d872 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:09.583164 1900 consensus_queue.cc:1035] T 88f696d03330448d879ed7b60364c28c P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:09.586274 29730 catalog_manager.cc:5576] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719) does not have the latest schema for tablet dd90eef09b7343a09cfda4554e67d872 (table rebalancer_dynamic_table [id=b7ee08c198394bb4a152be45d2d9199c]). Expected version 1 got 0
I20240503 02:13:09.588563 2530 tablet.cc:1722] T dd90eef09b7343a09cfda4554e67d872 P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:09.594007 2530 tablet.cc:1620] T dd90eef09b7343a09cfda4554e67d872 P f575ce92c0224eeb80b9fae5f6561fbe: MemRowSet was empty: no flush needed.
I20240503 02:13:09.596279 29731 catalog_manager.cc:5576] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241) does not have the latest schema for tablet dd90eef09b7343a09cfda4554e67d872 (table rebalancer_dynamic_table [id=b7ee08c198394bb4a152be45d2d9199c]). Expected version 1 got 0
I20240503 02:13:09.600761 2534 tablet.cc:1722] T dd90eef09b7343a09cfda4554e67d872 P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:09.603245 2534 tablet.cc:1620] T dd90eef09b7343a09cfda4554e67d872 P fbbc2be6382e47908ecb7c68c367bd26: MemRowSet was empty: no flush needed.
I20240503 02:13:09.603406 2412 consensus_queue.cc:1035] T 88f696d03330448d879ed7b60364c28c P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:09.609205 29731 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=b7ee08c198394bb4a152be45d2d9199c] alter complete (version 1)
I20240503 02:13:09.612149 2532 tablet.cc:1722] T dd90eef09b7343a09cfda4554e67d872 P 16440e365430405fb10366ab70e2b429: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:09.615243 2532 tablet.cc:1620] T dd90eef09b7343a09cfda4554e67d872 P 16440e365430405fb10366ab70e2b429: MemRowSet was empty: no flush needed.
I20240503 02:13:09.615811 29730 catalog_manager.cc:5576] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719) does not have the latest schema for tablet dd90eef09b7343a09cfda4554e67d872 (table rebalancer_dynamic_table [id=b7ee08c198394bb4a152be45d2d9199c]). Expected version 1 got 0
I20240503 02:13:09.653569 29730 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_extra_table_26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:09.668455 30116 tablet_service.cc:1467] Processing CreateTablet for tablet 83e8dad806564b128d5f0fe559d8ffe1 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=9a18661b2f054802a39965b2fb3ce514]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:09.668823 548 tablet_service.cc:1467] Processing CreateTablet for tablet 5800022ae50043ee8b1af1e134e86e6a (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=9a18661b2f054802a39965b2fb3ce514]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:09.669004 30116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83e8dad806564b128d5f0fe559d8ffe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:09.669293 548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5800022ae50043ee8b1af1e134e86e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:09.670570 32620 tablet_service.cc:1467] Processing CreateTablet for tablet 16cf978ac35341d4adea71a17a91194c (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=9a18661b2f054802a39965b2fb3ce514]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:09.670729 32619 tablet_service.cc:1467] Processing CreateTablet for tablet 83e8dad806564b128d5f0fe559d8ffe1 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=9a18661b2f054802a39965b2fb3ce514]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:09.670987 32620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16cf978ac35341d4adea71a17a91194c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:09.671340 32621 tablet_service.cc:1467] Processing CreateTablet for tablet 5800022ae50043ee8b1af1e134e86e6a (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=9a18661b2f054802a39965b2fb3ce514]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:09.671691 32621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5800022ae50043ee8b1af1e134e86e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:09.673375 415 tablet_service.cc:1467] Processing CreateTablet for tablet 83e8dad806564b128d5f0fe559d8ffe1 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=9a18661b2f054802a39965b2fb3ce514]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:09.673480 32755 tablet_service.cc:1467] Processing CreateTablet for tablet 16cf978ac35341d4adea71a17a91194c (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=9a18661b2f054802a39965b2fb3ce514]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:09.673835 415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83e8dad806564b128d5f0fe559d8ffe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:09.673914 32755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16cf978ac35341d4adea71a17a91194c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:09.677505 32754 tablet_service.cc:1467] Processing CreateTablet for tablet 5800022ae50043ee8b1af1e134e86e6a (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=9a18661b2f054802a39965b2fb3ce514]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:09.678016 32754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5800022ae50043ee8b1af1e134e86e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:09.681659 32619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83e8dad806564b128d5f0fe559d8ffe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:09.680923 416 tablet_service.cc:1467] Processing CreateTablet for tablet 16cf978ac35341d4adea71a17a91194c (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=9a18661b2f054802a39965b2fb3ce514]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:09.689594 416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16cf978ac35341d4adea71a17a91194c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:09.716524 29730 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:13:09.727681 2530 tablet.cc:1722] T dd90eef09b7343a09cfda4554e67d872 P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:09.728191 2530 tablet.cc:1620] T dd90eef09b7343a09cfda4554e67d872 P f575ce92c0224eeb80b9fae5f6561fbe: MemRowSet was empty: no flush needed.
I20240503 02:13:09.729681 2532 tablet.cc:1722] T dd90eef09b7343a09cfda4554e67d872 P 16440e365430405fb10366ab70e2b429: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:09.730206 2532 tablet.cc:1620] T dd90eef09b7343a09cfda4554e67d872 P 16440e365430405fb10366ab70e2b429: MemRowSet was empty: no flush needed.
I20240503 02:13:09.736224 2556 tablet_bootstrap.cc:492] T 83e8dad806564b128d5f0fe559d8ffe1 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:09.736811 2558 tablet_bootstrap.cc:492] T 83e8dad806564b128d5f0fe559d8ffe1 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:09.737923 2534 tablet.cc:1722] T dd90eef09b7343a09cfda4554e67d872 P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:09.738382 2534 tablet.cc:1620] T dd90eef09b7343a09cfda4554e67d872 P fbbc2be6382e47908ecb7c68c367bd26: MemRowSet was empty: no flush needed.
I20240503 02:13:09.746320 29734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=b7ee08c198394bb4a152be45d2d9199c] alter complete (version 2)
I20240503 02:13:09.752116 2557 tablet_bootstrap.cc:492] T 16cf978ac35341d4adea71a17a91194c P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:09.755231 2557 tablet_bootstrap.cc:654] T 16cf978ac35341d4adea71a17a91194c P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:09.756453 2558 tablet_bootstrap.cc:654] T 83e8dad806564b128d5f0fe559d8ffe1 P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:09.757741 2556 tablet_bootstrap.cc:654] T 83e8dad806564b128d5f0fe559d8ffe1 P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:09.760506 2553 tablet_bootstrap.cc:492] T 5800022ae50043ee8b1af1e134e86e6a P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:09.762805 2556 tablet_bootstrap.cc:492] T 83e8dad806564b128d5f0fe559d8ffe1 P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:09.763031 2556 ts_tablet_manager.cc:1367] T 83e8dad806564b128d5f0fe559d8ffe1 P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.002s
I20240503 02:13:09.763834 2553 tablet_bootstrap.cc:654] T 5800022ae50043ee8b1af1e134e86e6a P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:09.763974 2556 raft_consensus.cc:348] T 83e8dad806564b128d5f0fe559d8ffe1 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:09.764248 2556 raft_consensus.cc:374] T 83e8dad806564b128d5f0fe559d8ffe1 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:09.764386 2556 raft_consensus.cc:729] T 83e8dad806564b128d5f0fe559d8ffe1 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:09.766242 2556 consensus_queue.cc:260] T 83e8dad806564b128d5f0fe559d8ffe1 P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:09.770099 2556 ts_tablet_manager.cc:1398] T 83e8dad806564b128d5f0fe559d8ffe1 P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240503 02:13:09.770498 2556 tablet_bootstrap.cc:492] T 5800022ae50043ee8b1af1e134e86e6a P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:09.771255 2555 tablet_bootstrap.cc:492] T 83e8dad806564b128d5f0fe559d8ffe1 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:13:09.773082 2555 tablet_bootstrap.cc:654] T 83e8dad806564b128d5f0fe559d8ffe1 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:09.775223 2556 tablet_bootstrap.cc:654] T 5800022ae50043ee8b1af1e134e86e6a P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:09.778299 2558 tablet_bootstrap.cc:492] T 83e8dad806564b128d5f0fe559d8ffe1 P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:09.778659 2558 ts_tablet_manager.cc:1367] T 83e8dad806564b128d5f0fe559d8ffe1 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.042s user 0.008s sys 0.000s
I20240503 02:13:09.779996 2558 raft_consensus.cc:348] T 83e8dad806564b128d5f0fe559d8ffe1 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:09.780330 2558 raft_consensus.cc:374] T 83e8dad806564b128d5f0fe559d8ffe1 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:09.780478 2558 raft_consensus.cc:729] T 83e8dad806564b128d5f0fe559d8ffe1 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:09.780922 2558 consensus_queue.cc:260] T 83e8dad806564b128d5f0fe559d8ffe1 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:09.781965 2558 ts_tablet_manager.cc:1398] T 83e8dad806564b128d5f0fe559d8ffe1 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:13:09.783041 2557 tablet_bootstrap.cc:492] T 16cf978ac35341d4adea71a17a91194c P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:09.783222 2557 ts_tablet_manager.cc:1367] T 16cf978ac35341d4adea71a17a91194c P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.007s
I20240503 02:13:09.784090 2557 raft_consensus.cc:348] T 16cf978ac35341d4adea71a17a91194c P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:09.784300 2557 raft_consensus.cc:374] T 16cf978ac35341d4adea71a17a91194c P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:09.784387 2557 raft_consensus.cc:729] T 16cf978ac35341d4adea71a17a91194c P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:09.784663 2557 consensus_queue.cc:260] T 16cf978ac35341d4adea71a17a91194c P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:09.785653 2557 ts_tablet_manager.cc:1398] T 16cf978ac35341d4adea71a17a91194c P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:13:09.785948 2557 tablet_bootstrap.cc:492] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:09.788190 2557 tablet_bootstrap.cc:654] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:09.798878 2553 tablet_bootstrap.cc:492] T 5800022ae50043ee8b1af1e134e86e6a P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:09.798957 2558 tablet_bootstrap.cc:492] T 16cf978ac35341d4adea71a17a91194c P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:09.799152 2553 ts_tablet_manager.cc:1367] T 5800022ae50043ee8b1af1e134e86e6a P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.004s
I20240503 02:13:09.800005 2553 raft_consensus.cc:348] T 5800022ae50043ee8b1af1e134e86e6a P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:09.800390 2553 raft_consensus.cc:374] T 5800022ae50043ee8b1af1e134e86e6a P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:09.800554 2553 raft_consensus.cc:729] T 5800022ae50043ee8b1af1e134e86e6a P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:09.801219 2558 tablet_bootstrap.cc:654] T 16cf978ac35341d4adea71a17a91194c P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:09.800884 2553 consensus_queue.cc:260] T 5800022ae50043ee8b1af1e134e86e6a P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:09.811231 1909 raft_consensus.cc:482] T 5800022ae50043ee8b1af1e134e86e6a P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:09.811478 1909 raft_consensus.cc:504] T 5800022ae50043ee8b1af1e134e86e6a P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:09.812233 1909 leader_election.cc:288] T 5800022ae50043ee8b1af1e134e86e6a P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:09.812788 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5800022ae50043ee8b1af1e134e86e6a" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:13:09.814404 2553 ts_tablet_manager.cc:1398] T 5800022ae50043ee8b1af1e134e86e6a P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240503 02:13:09.815599 2556 tablet_bootstrap.cc:492] T 5800022ae50043ee8b1af1e134e86e6a P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:09.815821 2556 ts_tablet_manager.cc:1367] T 5800022ae50043ee8b1af1e134e86e6a P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.045s user 0.000s sys 0.005s
I20240503 02:13:09.816802 2556 raft_consensus.cc:348] T 5800022ae50043ee8b1af1e134e86e6a P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:09.817080 2556 raft_consensus.cc:374] T 5800022ae50043ee8b1af1e134e86e6a P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:09.817160 2556 raft_consensus.cc:729] T 5800022ae50043ee8b1af1e134e86e6a P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:09.817394 2556 consensus_queue.cc:260] T 5800022ae50043ee8b1af1e134e86e6a P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
W20240503 02:13:09.817569 502 leader_election.cc:341] T 5800022ae50043ee8b1af1e134e86e6a P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:09.819151 2556 ts_tablet_manager.cc:1398] T 5800022ae50043ee8b1af1e134e86e6a P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:09.819518 2556 tablet_bootstrap.cc:492] T 16cf978ac35341d4adea71a17a91194c P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:09.820756 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5800022ae50043ee8b1af1e134e86e6a" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:13:09.821044 32641 raft_consensus.cc:2454] T 5800022ae50043ee8b1af1e134e86e6a P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 0.
I20240503 02:13:09.821676 2558 tablet_bootstrap.cc:492] T 16cf978ac35341d4adea71a17a91194c P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:09.821925 2558 ts_tablet_manager.cc:1367] T 16cf978ac35341d4adea71a17a91194c P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.009s
I20240503 02:13:09.822108 2556 tablet_bootstrap.cc:654] T 16cf978ac35341d4adea71a17a91194c P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:09.822868 502 leader_election.cc:302] T 5800022ae50043ee8b1af1e134e86e6a P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe; no voters: 8a666e798fe8477d8dd1e51b385d5ead
I20240503 02:13:09.823067 2558 raft_consensus.cc:348] T 16cf978ac35341d4adea71a17a91194c P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:09.823433 1909 raft_consensus.cc:2781] T 5800022ae50043ee8b1af1e134e86e6a P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:09.823544 1909 raft_consensus.cc:482] T 5800022ae50043ee8b1af1e134e86e6a P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:09.823673 1909 raft_consensus.cc:3037] T 5800022ae50043ee8b1af1e134e86e6a P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:09.823714 2558 raft_consensus.cc:374] T 16cf978ac35341d4adea71a17a91194c P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:09.823935 2558 raft_consensus.cc:729] T 16cf978ac35341d4adea71a17a91194c P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:09.824187 2555 tablet_bootstrap.cc:492] T 83e8dad806564b128d5f0fe559d8ffe1 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:13:09.824280 2558 consensus_queue.cc:260] T 16cf978ac35341d4adea71a17a91194c P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:09.824476 2555 ts_tablet_manager.cc:1367] T 83e8dad806564b128d5f0fe559d8ffe1 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.053s user 0.008s sys 0.004s
I20240503 02:13:09.826187 2555 raft_consensus.cc:348] T 83e8dad806564b128d5f0fe559d8ffe1 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:09.826833 1909 raft_consensus.cc:504] T 5800022ae50043ee8b1af1e134e86e6a P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:09.827667 2555 raft_consensus.cc:374] T 83e8dad806564b128d5f0fe559d8ffe1 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:09.827975 2555 raft_consensus.cc:729] T 83e8dad806564b128d5f0fe559d8ffe1 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:13:09.829063 2557 tablet_bootstrap.cc:492] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:09.829310 2557 ts_tablet_manager.cc:1367] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.043s user 0.001s sys 0.011s
I20240503 02:13:09.828809 2555 consensus_queue.cc:260] T 83e8dad806564b128d5f0fe559d8ffe1 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:09.830797 2555 ts_tablet_manager.cc:1398] T 83e8dad806564b128d5f0fe559d8ffe1 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 02:13:09.831880 2558 ts_tablet_manager.cc:1398] T 16cf978ac35341d4adea71a17a91194c P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 02:13:09.833225 2556 tablet_bootstrap.cc:492] T 16cf978ac35341d4adea71a17a91194c P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:09.833384 2556 ts_tablet_manager.cc:1367] T 16cf978ac35341d4adea71a17a91194c P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20240503 02:13:09.834156 2556 raft_consensus.cc:348] T 16cf978ac35341d4adea71a17a91194c P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:09.834357 2556 raft_consensus.cc:374] T 16cf978ac35341d4adea71a17a91194c P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:09.834447 2556 raft_consensus.cc:729] T 16cf978ac35341d4adea71a17a91194c P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:09.834738 2556 consensus_queue.cc:260] T 16cf978ac35341d4adea71a17a91194c P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:09.835034 2557 raft_consensus.cc:348] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:09.835249 2557 raft_consensus.cc:374] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:09.835310 2557 raft_consensus.cc:729] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:09.835391 2556 ts_tablet_manager.cc:1398] T 16cf978ac35341d4adea71a17a91194c P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:09.835592 2557 consensus_queue.cc:260] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:09.839195 2557 ts_tablet_manager.cc:1398] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 02:13:09.840428 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5800022ae50043ee8b1af1e134e86e6a" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d"
I20240503 02:13:09.840632 32641 raft_consensus.cc:3037] T 5800022ae50043ee8b1af1e134e86e6a P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:09.840550 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5800022ae50043ee8b1af1e134e86e6a" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:13:09.841181 307 raft_consensus.cc:3037] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:09.842824 32641 raft_consensus.cc:2454] T 5800022ae50043ee8b1af1e134e86e6a P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 1.
I20240503 02:13:09.843259 502 leader_election.cc:302] T 5800022ae50043ee8b1af1e134e86e6a P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:09.843586 2494 raft_consensus.cc:2781] T 5800022ae50043ee8b1af1e134e86e6a P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:09.843717 2494 raft_consensus.cc:686] T 5800022ae50043ee8b1af1e134e86e6a P f575ce92c0224eeb80b9fae5f6561fbe [term 1 LEADER]: Becoming Leader. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Running, Role: LEADER
I20240503 02:13:09.843917 307 raft_consensus.cc:2454] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 1.
I20240503 02:13:09.844012 2494 consensus_queue.cc:237] T 5800022ae50043ee8b1af1e134e86e6a P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:09.845444 1909 leader_election.cc:288] T 5800022ae50043ee8b1af1e134e86e6a P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 election: Requested vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:09.859912 29725 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:13:09.861286 29734 catalog_manager.cc:5484] T 5800022ae50043ee8b1af1e134e86e6a P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: term changed from 0 to 1, leader changed from <none> to f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8). New cstate: current_term: 1 leader_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:09.876749 2177 raft_consensus.cc:482] T 83e8dad806564b128d5f0fe559d8ffe1 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:09.876999 2177 raft_consensus.cc:504] T 83e8dad806564b128d5f0fe559d8ffe1 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:09.878064 2177 leader_election.cc:288] T 83e8dad806564b128d5f0fe559d8ffe1 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:13:09.883849 30135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83e8dad806564b128d5f0fe559d8ffe1" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:13:09.884085 30135 raft_consensus.cc:2454] T 83e8dad806564b128d5f0fe559d8ffe1 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 0.
I20240503 02:13:09.889678 370 leader_election.cc:302] T 83e8dad806564b128d5f0fe559d8ffe1 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:09.890337 1900 raft_consensus.cc:2781] T 83e8dad806564b128d5f0fe559d8ffe1 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:09.890447 1900 raft_consensus.cc:482] T 83e8dad806564b128d5f0fe559d8ffe1 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:09.890558 1900 raft_consensus.cc:3037] T 83e8dad806564b128d5f0fe559d8ffe1 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:09.893697 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83e8dad806564b128d5f0fe559d8ffe1" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:13:09.893970 32641 raft_consensus.cc:2454] T 83e8dad806564b128d5f0fe559d8ffe1 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 0.
I20240503 02:13:09.895563 2530 tablet.cc:1722] T dd90eef09b7343a09cfda4554e67d872 P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:09.896736 2532 tablet.cc:1722] T dd90eef09b7343a09cfda4554e67d872 P 16440e365430405fb10366ab70e2b429: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:09.901909 2534 tablet.cc:1722] T dd90eef09b7343a09cfda4554e67d872 P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:09.911957 29730 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table_new_name [id=b7ee08c198394bb4a152be45d2d9199c] alter complete (version 3)
I20240503 02:13:09.912608 1900 raft_consensus.cc:504] T 83e8dad806564b128d5f0fe559d8ffe1 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:09.913925 1900 leader_election.cc:288] T 83e8dad806564b128d5f0fe559d8ffe1 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Requested vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:13:09.915216 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83e8dad806564b128d5f0fe559d8ffe1" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d"
I20240503 02:13:09.915493 32640 raft_consensus.cc:3037] T 83e8dad806564b128d5f0fe559d8ffe1 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:09.919195 32640 raft_consensus.cc:2454] T 83e8dad806564b128d5f0fe559d8ffe1 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 1.
I20240503 02:13:09.920099 30135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83e8dad806564b128d5f0fe559d8ffe1" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:13:09.920329 30135 raft_consensus.cc:3037] T 83e8dad806564b128d5f0fe559d8ffe1 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:09.920586 371 leader_election.cc:302] T 83e8dad806564b128d5f0fe559d8ffe1 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:09.921133 2177 raft_consensus.cc:2781] T 83e8dad806564b128d5f0fe559d8ffe1 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:09.921288 2177 raft_consensus.cc:686] T 83e8dad806564b128d5f0fe559d8ffe1 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Becoming Leader. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Running, Role: LEADER
I20240503 02:13:09.921548 2177 consensus_queue.cc:237] T 83e8dad806564b128d5f0fe559d8ffe1 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:09.923103 30135 raft_consensus.cc:2454] T 83e8dad806564b128d5f0fe559d8ffe1 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 1.
I20240503 02:13:09.949646 29725 catalog_manager.cc:5484] T 83e8dad806564b128d5f0fe559d8ffe1 P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7). New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:09.967633 1787 raft_consensus.cc:482] T 16cf978ac35341d4adea71a17a91194c P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:09.967844 1787 raft_consensus.cc:504] T 16cf978ac35341d4adea71a17a91194c P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:09.968484 1787 leader_election.cc:288] T 16cf978ac35341d4adea71a17a91194c P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:09.969467 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16cf978ac35341d4adea71a17a91194c" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:13:09.969712 307 raft_consensus.cc:2454] T 16cf978ac35341d4adea71a17a91194c P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 0.
I20240503 02:13:09.970244 32574 leader_election.cc:302] T 16cf978ac35341d4adea71a17a91194c P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, 9a4d92bc25374fb3960fde7d10be309d; no voters:
I20240503 02:13:09.970728 1787 raft_consensus.cc:2781] T 16cf978ac35341d4adea71a17a91194c P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:09.970829 1787 raft_consensus.cc:482] T 16cf978ac35341d4adea71a17a91194c P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:09.970932 1787 raft_consensus.cc:3037] T 16cf978ac35341d4adea71a17a91194c P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:09.973788 1787 raft_consensus.cc:504] T 16cf978ac35341d4adea71a17a91194c P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:09.974900 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16cf978ac35341d4adea71a17a91194c" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:13:09.975085 307 raft_consensus.cc:3037] T 16cf978ac35341d4adea71a17a91194c P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:09.978289 307 raft_consensus.cc:2454] T 16cf978ac35341d4adea71a17a91194c P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 1.
I20240503 02:13:09.980634 32574 leader_election.cc:302] T 16cf978ac35341d4adea71a17a91194c P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, 9a4d92bc25374fb3960fde7d10be309d; no voters:
I20240503 02:13:09.981796 1787 leader_election.cc:288] T 16cf978ac35341d4adea71a17a91194c P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 election: Requested vote from peers 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:09.982297 1787 raft_consensus.cc:2781] T 16cf978ac35341d4adea71a17a91194c P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:09.982724 1787 raft_consensus.cc:686] T 16cf978ac35341d4adea71a17a91194c P 9a4d92bc25374fb3960fde7d10be309d [term 1 LEADER]: Becoming Leader. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Running, Role: LEADER
I20240503 02:13:09.983238 1787 consensus_queue.cc:237] T 16cf978ac35341d4adea71a17a91194c P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:09.992484 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16cf978ac35341d4adea71a17a91194c" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:13:09.992892 437 raft_consensus.cc:2454] T 16cf978ac35341d4adea71a17a91194c P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 0.
I20240503 02:13:10.009438 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16cf978ac35341d4adea71a17a91194c" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:13:10.009675 437 raft_consensus.cc:3037] T 16cf978ac35341d4adea71a17a91194c P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:10.013171 437 raft_consensus.cc:2454] T 16cf978ac35341d4adea71a17a91194c P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 1.
I20240503 02:13:10.023051 29730 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:10.023294 29730 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:10.031704 29725 catalog_manager.cc:5484] T 16cf978ac35341d4adea71a17a91194c P 9a4d92bc25374fb3960fde7d10be309d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5). New cstate: current_term: 1 leader_uuid: "9a4d92bc25374fb3960fde7d10be309d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:10.038512 29730 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet dd90eef09b7343a09cfda4554e67d872
I20240503 02:13:10.039463 29853 tablet_service.cc:1514] Processing DeleteTablet for tablet dd90eef09b7343a09cfda4554e67d872 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:10 UTC) from {username='slave'} at 127.0.0.1:58744
I20240503 02:13:10.046195 548 tablet_service.cc:1514] Processing DeleteTablet for tablet dd90eef09b7343a09cfda4554e67d872 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:10 UTC) from {username='slave'} at 127.0.0.1:39028
I20240503 02:13:10.046980 415 tablet_service.cc:1514] Processing DeleteTablet for tablet dd90eef09b7343a09cfda4554e67d872 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:10 UTC) from {username='slave'} at 127.0.0.1:50814
I20240503 02:13:10.059684 2586 tablet_replica.cc:330] T dd90eef09b7343a09cfda4554e67d872 P fbbc2be6382e47908ecb7c68c367bd26: stopping tablet replica
I20240503 02:13:10.059002 2584 tablet_replica.cc:330] T dd90eef09b7343a09cfda4554e67d872 P f575ce92c0224eeb80b9fae5f6561fbe: stopping tablet replica
I20240503 02:13:10.060118 2584 raft_consensus.cc:2229] T dd90eef09b7343a09cfda4554e67d872 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:10.060391 2586 raft_consensus.cc:2229] T dd90eef09b7343a09cfda4554e67d872 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:10.060808 2584 raft_consensus.cc:2258] T dd90eef09b7343a09cfda4554e67d872 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:10.061112 2586 raft_consensus.cc:2258] T dd90eef09b7343a09cfda4554e67d872 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:10.063115 2584 ts_tablet_manager.cc:1875] T dd90eef09b7343a09cfda4554e67d872 P f575ce92c0224eeb80b9fae5f6561fbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:10.063772 2586 ts_tablet_manager.cc:1875] T dd90eef09b7343a09cfda4554e67d872 P fbbc2be6382e47908ecb7c68c367bd26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:10.072315 2583 tablet_replica.cc:330] T dd90eef09b7343a09cfda4554e67d872 P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:13:10.073180 2583 raft_consensus.cc:2229] T dd90eef09b7343a09cfda4554e67d872 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:10.073493 2583 raft_consensus.cc:2258] T dd90eef09b7343a09cfda4554e67d872 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:10.074589 2583 ts_tablet_manager.cc:1875] T dd90eef09b7343a09cfda4554e67d872 P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:10.080039 2584 ts_tablet_manager.cc:1888] T dd90eef09b7343a09cfda4554e67d872 P f575ce92c0224eeb80b9fae5f6561fbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:10.080493 2586 ts_tablet_manager.cc:1888] T dd90eef09b7343a09cfda4554e67d872 P fbbc2be6382e47908ecb7c68c367bd26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:10.080739 2586 log.cc:1196] T dd90eef09b7343a09cfda4554e67d872 P fbbc2be6382e47908ecb7c68c367bd26: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal/wals/dd90eef09b7343a09cfda4554e67d872
I20240503 02:13:10.081010 2584 log.cc:1196] T dd90eef09b7343a09cfda4554e67d872 P f575ce92c0224eeb80b9fae5f6561fbe: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal/wals/dd90eef09b7343a09cfda4554e67d872
I20240503 02:13:10.081389 2583 ts_tablet_manager.cc:1888] T dd90eef09b7343a09cfda4554e67d872 P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:10.081583 2583 log.cc:1196] T dd90eef09b7343a09cfda4554e67d872 P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/dd90eef09b7343a09cfda4554e67d872
I20240503 02:13:10.081894 2584 ts_tablet_manager.cc:1909] T dd90eef09b7343a09cfda4554e67d872 P f575ce92c0224eeb80b9fae5f6561fbe: Deleting consensus metadata
I20240503 02:13:10.082139 2583 ts_tablet_manager.cc:1909] T dd90eef09b7343a09cfda4554e67d872 P 16440e365430405fb10366ab70e2b429: Deleting consensus metadata
I20240503 02:13:10.082293 2586 ts_tablet_manager.cc:1909] T dd90eef09b7343a09cfda4554e67d872 P fbbc2be6382e47908ecb7c68c367bd26: Deleting consensus metadata
I20240503 02:13:10.083325 29719 catalog_manager.cc:4830] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): tablet dd90eef09b7343a09cfda4554e67d872 (table rebalancer_dynamic_table_new_name [id=b7ee08c198394bb4a152be45d2d9199c]) successfully deleted
I20240503 02:13:10.084349 29718 catalog_manager.cc:4830] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): tablet dd90eef09b7343a09cfda4554e67d872 (table rebalancer_dynamic_table_new_name [id=b7ee08c198394bb4a152be45d2d9199c]) successfully deleted
I20240503 02:13:10.086181 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet dd90eef09b7343a09cfda4554e67d872 (table rebalancer_dynamic_table_new_name [id=b7ee08c198394bb4a152be45d2d9199c]) successfully deleted
I20240503 02:13:10.136507 30245 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:10.142792 29984 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:13:10.148298 29730 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:13:10.151156 29853 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:10.159976 32755 tablet_service.cc:1467] Processing CreateTablet for tablet f0766669c2494c7db1590730b2c163ca (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9c822d1fd0454726a711868b2428a47c]), partition=
I20240503 02:13:10.160452 32755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0766669c2494c7db1590730b2c163ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:10.160450 32620 tablet_service.cc:1467] Processing CreateTablet for tablet f0766669c2494c7db1590730b2c163ca (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9c822d1fd0454726a711868b2428a47c]), partition=
I20240503 02:13:10.160820 32620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0766669c2494c7db1590730b2c163ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:10.166148 548 tablet_service.cc:1467] Processing CreateTablet for tablet f0766669c2494c7db1590730b2c163ca (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9c822d1fd0454726a711868b2428a47c]), partition=
I20240503 02:13:10.166565 548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0766669c2494c7db1590730b2c163ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:10.169440 2556 tablet_bootstrap.cc:492] T f0766669c2494c7db1590730b2c163ca P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:10.171360 2556 tablet_bootstrap.cc:654] T f0766669c2494c7db1590730b2c163ca P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:10.186456 30116 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 02:13:10.194464 32708 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.8:42233, user_credentials={real_user=slave}} blocked reactor thread for 48725.9us
I20240503 02:13:10.217310 2557 tablet_bootstrap.cc:492] T f0766669c2494c7db1590730b2c163ca P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:10.219046 2557 tablet_bootstrap.cc:654] T f0766669c2494c7db1590730b2c163ca P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:10.220176 32754 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:10.230970 2553 tablet_bootstrap.cc:492] T f0766669c2494c7db1590730b2c163ca P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:10.232784 2553 tablet_bootstrap.cc:654] T f0766669c2494c7db1590730b2c163ca P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:10.237571 2494 consensus_queue.cc:1035] T 5800022ae50043ee8b1af1e134e86e6a P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:10.247658 415 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:13:10.249383 2557 tablet_bootstrap.cc:492] T f0766669c2494c7db1590730b2c163ca P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:10.249666 2557 ts_tablet_manager.cc:1367] T f0766669c2494c7db1590730b2c163ca P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.001s
I20240503 02:13:10.250681 2557 raft_consensus.cc:348] T f0766669c2494c7db1590730b2c163ca P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:10.256270 2557 raft_consensus.cc:374] T f0766669c2494c7db1590730b2c163ca P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:10.256763 2557 raft_consensus.cc:729] T f0766669c2494c7db1590730b2c163ca P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:10.261552 2557 consensus_queue.cc:260] T f0766669c2494c7db1590730b2c163ca P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:10.262403 2557 ts_tablet_manager.cc:1398] T f0766669c2494c7db1590730b2c163ca P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240503 02:13:10.281956 2556 tablet_bootstrap.cc:492] T f0766669c2494c7db1590730b2c163ca P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:10.283175 2556 ts_tablet_manager.cc:1367] T f0766669c2494c7db1590730b2c163ca P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.114s user 0.005s sys 0.004s
I20240503 02:13:10.289273 2494 consensus_queue.cc:1035] T 5800022ae50043ee8b1af1e134e86e6a P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:10.310612 2553 tablet_bootstrap.cc:492] T f0766669c2494c7db1590730b2c163ca P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:10.310907 2553 ts_tablet_manager.cc:1367] T f0766669c2494c7db1590730b2c163ca P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.080s user 0.002s sys 0.008s
I20240503 02:13:10.311834 2553 raft_consensus.cc:348] T f0766669c2494c7db1590730b2c163ca P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:10.312157 2553 raft_consensus.cc:374] T f0766669c2494c7db1590730b2c163ca P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:10.312278 2553 raft_consensus.cc:729] T f0766669c2494c7db1590730b2c163ca P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:10.312639 2553 consensus_queue.cc:260] T f0766669c2494c7db1590730b2c163ca P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:10.313429 2553 ts_tablet_manager.cc:1398] T f0766669c2494c7db1590730b2c163ca P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:13:10.318864 2556 raft_consensus.cc:348] T f0766669c2494c7db1590730b2c163ca P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:10.322052 2556 raft_consensus.cc:374] T f0766669c2494c7db1590730b2c163ca P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:10.322173 2556 raft_consensus.cc:729] T f0766669c2494c7db1590730b2c163ca P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:10.322438 2556 consensus_queue.cc:260] T f0766669c2494c7db1590730b2c163ca P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:10.331240 2556 ts_tablet_manager.cc:1398] T f0766669c2494c7db1590730b2c163ca P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.047s user 0.001s sys 0.001s
I20240503 02:13:10.335476 2177 consensus_queue.cc:1035] T 83e8dad806564b128d5f0fe559d8ffe1 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:10.337401 548 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:10.340387 32620 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:13:10.377836 2412 consensus_queue.cc:1035] T 83e8dad806564b128d5f0fe559d8ffe1 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:10.395584 1787 consensus_queue.cc:1035] T 16cf978ac35341d4adea71a17a91194c P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 02:13:10.418629 2580 consensus_queue.cc:1035] T 16cf978ac35341d4adea71a17a91194c P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:10.499640 29874 consensus_queue.cc:237] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:10.501659 30134 raft_consensus.cc:1261] T 32a2dbf80daf40d6b21442a39f62f14c P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:10.501673 307 raft_consensus.cc:1261] T 32a2dbf80daf40d6b21442a39f62f14c P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:10.502283 2027 consensus_queue.cc:1035] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:10.502820 1484 consensus_queue.cc:1035] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:10.507443 2027 raft_consensus.cc:2932] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:10.508248 30134 raft_consensus.cc:2932] T 32a2dbf80daf40d6b21442a39f62f14c P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:10.514868 29732 catalog_manager.cc:5484] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 reported cstate change: config changed from index 4 to 5, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) evicted. New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:10.519980 30245 tablet_service.cc:1514] Processing DeleteTablet for tablet 32a2dbf80daf40d6b21442a39f62f14c with delete_type TABLET_DATA_TOMBSTONED (TS 1f10f80f13c84e80b03eda7085a3b012 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:49856
I20240503 02:13:10.520680 2644 tablet_replica.cc:330] T 32a2dbf80daf40d6b21442a39f62f14c P 1f10f80f13c84e80b03eda7085a3b012: stopping tablet replica
I20240503 02:13:10.521149 2644 raft_consensus.cc:2229] T 32a2dbf80daf40d6b21442a39f62f14c P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:10.521791 2644 raft_consensus.cc:2258] T 32a2dbf80daf40d6b21442a39f62f14c P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:10.522891 2644 ts_tablet_manager.cc:1875] T 32a2dbf80daf40d6b21442a39f62f14c P 1f10f80f13c84e80b03eda7085a3b012: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:10.524564 307 raft_consensus.cc:2932] T 32a2dbf80daf40d6b21442a39f62f14c P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:10.532109 2644 ts_tablet_manager.cc:1888] T 32a2dbf80daf40d6b21442a39f62f14c P 1f10f80f13c84e80b03eda7085a3b012: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:13:10.532371 2644 log.cc:1196] T 32a2dbf80daf40d6b21442a39f62f14c P 1f10f80f13c84e80b03eda7085a3b012: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/wals/32a2dbf80daf40d6b21442a39f62f14c
I20240503 02:13:10.533425 29719 catalog_manager.cc:4830] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): tablet 32a2dbf80daf40d6b21442a39f62f14c (table rebalance_test_table_17 [id=6fdcf94cbd1844948615de6c2a4d18cc]) successfully deleted
I20240503 02:13:10.587821 29730 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_extra_table_28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:10.598605 30245 tablet_service.cc:1467] Processing CreateTablet for tablet 7263a253c5884bce987d28e188ccca11 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=d70cb2e6817a47df8925321a6a778740]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:10.598965 415 tablet_service.cc:1467] Processing CreateTablet for tablet 7263a253c5884bce987d28e188ccca11 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=d70cb2e6817a47df8925321a6a778740]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:10.599133 30245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7263a253c5884bce987d28e188ccca11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:10.599411 415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7263a253c5884bce987d28e188ccca11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:10.599368 32620 tablet_service.cc:1467] Processing CreateTablet for tablet 7263a253c5884bce987d28e188ccca11 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=d70cb2e6817a47df8925321a6a778740]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:10.599803 32620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7263a253c5884bce987d28e188ccca11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:10.600912 548 tablet_service.cc:1467] Processing CreateTablet for tablet ac4d497de05c49edbd6e0edac73c058d (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=d70cb2e6817a47df8925321a6a778740]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:10.601390 548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac4d497de05c49edbd6e0edac73c058d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:10.602097 416 tablet_service.cc:1467] Processing CreateTablet for tablet ac4d497de05c49edbd6e0edac73c058d (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=d70cb2e6817a47df8925321a6a778740]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:10.602491 416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac4d497de05c49edbd6e0edac73c058d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:10.612453 32755 tablet_service.cc:1467] Processing CreateTablet for tablet ac4d497de05c49edbd6e0edac73c058d (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=d70cb2e6817a47df8925321a6a778740]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:10.612689 32619 tablet_service.cc:1467] Processing CreateTablet for tablet 08e4d119d90f4146ba53a1b18efe2386 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=d70cb2e6817a47df8925321a6a778740]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:10.612969 32755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac4d497de05c49edbd6e0edac73c058d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:10.614145 417 tablet_service.cc:1467] Processing CreateTablet for tablet 08e4d119d90f4146ba53a1b18efe2386 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=d70cb2e6817a47df8925321a6a778740]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:10.614714 417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08e4d119d90f4146ba53a1b18efe2386. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:10.617573 32754 tablet_service.cc:1467] Processing CreateTablet for tablet 08e4d119d90f4146ba53a1b18efe2386 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=d70cb2e6817a47df8925321a6a778740]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:10.618029 32754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08e4d119d90f4146ba53a1b18efe2386. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:10.619326 2666 tablet_bootstrap.cc:492] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:10.620383 32619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08e4d119d90f4146ba53a1b18efe2386. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:10.621052 2666 tablet_bootstrap.cc:654] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:10.621208 2553 tablet_bootstrap.cc:492] T ac4d497de05c49edbd6e0edac73c058d P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:10.622920 2665 tablet_bootstrap.cc:492] T 7263a253c5884bce987d28e188ccca11 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:13:10.623427 2553 tablet_bootstrap.cc:654] T ac4d497de05c49edbd6e0edac73c058d P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:10.627526 2665 tablet_bootstrap.cc:654] T 7263a253c5884bce987d28e188ccca11 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:10.635237 2557 tablet_bootstrap.cc:492] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:10.636399 2666 tablet_bootstrap.cc:492] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:10.636660 2666 ts_tablet_manager.cc:1367] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.000s
I20240503 02:13:10.637679 2557 tablet_bootstrap.cc:654] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:10.637601 2666 raft_consensus.cc:348] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:10.637987 2666 raft_consensus.cc:374] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:10.638139 2666 raft_consensus.cc:729] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:10.638487 2666 consensus_queue.cc:260] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:10.658121 2604 raft_consensus.cc:482] T f0766669c2494c7db1590730b2c163ca P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:10.658303 2604 raft_consensus.cc:504] T f0766669c2494c7db1590730b2c163ca P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:10.659145 2604 leader_election.cc:288] T f0766669c2494c7db1590730b2c163ca P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:13:10.660403 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0766669c2494c7db1590730b2c163ca" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:13:10.660660 32641 raft_consensus.cc:2454] T f0766669c2494c7db1590730b2c163ca P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:13:10.661170 32709 leader_election.cc:302] T f0766669c2494c7db1590730b2c163ca P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, 9a4d92bc25374fb3960fde7d10be309d; no voters:
I20240503 02:13:10.661491 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0766669c2494c7db1590730b2c163ca" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:10.661742 567 raft_consensus.cc:2454] T f0766669c2494c7db1590730b2c163ca P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:13:10.662963 2604 raft_consensus.cc:2781] T f0766669c2494c7db1590730b2c163ca P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:10.663100 2604 raft_consensus.cc:482] T f0766669c2494c7db1590730b2c163ca P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:10.663201 2604 raft_consensus.cc:3037] T f0766669c2494c7db1590730b2c163ca P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:10.669831 2556 tablet_bootstrap.cc:492] T 7263a253c5884bce987d28e188ccca11 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:10.669941 2604 raft_consensus.cc:504] T f0766669c2494c7db1590730b2c163ca P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:10.670526 2553 tablet_bootstrap.cc:492] T ac4d497de05c49edbd6e0edac73c058d P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:10.670734 2553 ts_tablet_manager.cc:1367] T ac4d497de05c49edbd6e0edac73c058d P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.050s user 0.008s sys 0.004s
I20240503 02:13:10.670998 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0766669c2494c7db1590730b2c163ca" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d"
I20240503 02:13:10.671227 32641 raft_consensus.cc:3037] T f0766669c2494c7db1590730b2c163ca P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:10.671648 2556 tablet_bootstrap.cc:654] T 7263a253c5884bce987d28e188ccca11 P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:10.671756 2553 raft_consensus.cc:348] T ac4d497de05c49edbd6e0edac73c058d P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:10.672021 2553 raft_consensus.cc:374] T ac4d497de05c49edbd6e0edac73c058d P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:10.672123 2553 raft_consensus.cc:729] T ac4d497de05c49edbd6e0edac73c058d P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:10.674386 2557 tablet_bootstrap.cc:492] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:10.674623 2557 ts_tablet_manager.cc:1367] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.039s user 0.008s sys 0.001s
I20240503 02:13:10.675560 2557 raft_consensus.cc:348] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:10.675794 2557 raft_consensus.cc:374] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:10.675870 2557 raft_consensus.cc:729] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:10.676050 32641 raft_consensus.cc:2454] T f0766669c2494c7db1590730b2c163ca P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:13:10.676139 2557 consensus_queue.cc:260] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:10.676517 32709 leader_election.cc:302] T f0766669c2494c7db1590730b2c163ca P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, 9a4d92bc25374fb3960fde7d10be309d; no voters:
I20240503 02:13:10.676753 2557 ts_tablet_manager.cc:1398] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:13:10.676921 2605 raft_consensus.cc:2781] T f0766669c2494c7db1590730b2c163ca P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:10.677065 2605 raft_consensus.cc:686] T f0766669c2494c7db1590730b2c163ca P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Becoming Leader. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Running, Role: LEADER
I20240503 02:13:10.677176 2557 tablet_bootstrap.cc:492] T 08e4d119d90f4146ba53a1b18efe2386 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:10.677400 2605 consensus_queue.cc:237] T f0766669c2494c7db1590730b2c163ca P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:10.678835 2553 consensus_queue.cc:260] T ac4d497de05c49edbd6e0edac73c058d P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:10.690191 2553 ts_tablet_manager.cc:1398] T ac4d497de05c49edbd6e0edac73c058d P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.019s user 0.002s sys 0.002s
I20240503 02:13:10.690613 2604 leader_election.cc:288] T f0766669c2494c7db1590730b2c163ca P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Requested vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:13:10.691463 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0766669c2494c7db1590730b2c163ca" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:13:10.691699 567 raft_consensus.cc:3037] T f0766669c2494c7db1590730b2c163ca P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:10.694286 2557 tablet_bootstrap.cc:654] T 08e4d119d90f4146ba53a1b18efe2386 P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:10.695199 567 raft_consensus.cc:2454] T f0766669c2494c7db1590730b2c163ca P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:13:10.639623 2666 ts_tablet_manager.cc:1398] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:10.702003 2666 tablet_bootstrap.cc:492] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:10.709129 2666 tablet_bootstrap.cc:654] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:10.716960 2556 tablet_bootstrap.cc:492] T 7263a253c5884bce987d28e188ccca11 P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:10.717204 2556 ts_tablet_manager.cc:1367] T 7263a253c5884bce987d28e188ccca11 P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.047s user 0.004s sys 0.005s
I20240503 02:13:10.718226 2556 raft_consensus.cc:348] T 7263a253c5884bce987d28e188ccca11 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:10.718454 2556 raft_consensus.cc:374] T 7263a253c5884bce987d28e188ccca11 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:10.718557 2556 raft_consensus.cc:729] T 7263a253c5884bce987d28e188ccca11 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:10.719426 2556 consensus_queue.cc:260] T 7263a253c5884bce987d28e188ccca11 P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:10.720350 2556 ts_tablet_manager.cc:1398] T 7263a253c5884bce987d28e188ccca11 P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240503 02:13:10.721191 2556 tablet_bootstrap.cc:492] T 08e4d119d90f4146ba53a1b18efe2386 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:10.723448 2556 tablet_bootstrap.cc:654] T 08e4d119d90f4146ba53a1b18efe2386 P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:10.724650 2557 tablet_bootstrap.cc:492] T 08e4d119d90f4146ba53a1b18efe2386 P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:10.724843 2557 ts_tablet_manager.cc:1367] T 08e4d119d90f4146ba53a1b18efe2386 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.048s user 0.005s sys 0.015s
I20240503 02:13:10.725927 2557 raft_consensus.cc:348] T 08e4d119d90f4146ba53a1b18efe2386 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:10.726224 2557 raft_consensus.cc:374] T 08e4d119d90f4146ba53a1b18efe2386 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:10.726312 2557 raft_consensus.cc:729] T 08e4d119d90f4146ba53a1b18efe2386 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:10.726413 2665 tablet_bootstrap.cc:492] T 7263a253c5884bce987d28e188ccca11 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:13:10.726755 2665 ts_tablet_manager.cc:1367] T 7263a253c5884bce987d28e188ccca11 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.104s user 0.003s sys 0.013s
I20240503 02:13:10.726604 2557 consensus_queue.cc:260] T 08e4d119d90f4146ba53a1b18efe2386 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:10.727357 2557 ts_tablet_manager.cc:1398] T 08e4d119d90f4146ba53a1b18efe2386 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:13:10.729357 2665 raft_consensus.cc:348] T 7263a253c5884bce987d28e188ccca11 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:10.732242 2665 raft_consensus.cc:374] T 7263a253c5884bce987d28e188ccca11 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:10.733105 2665 raft_consensus.cc:729] T 7263a253c5884bce987d28e188ccca11 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:13:10.736125 2605 raft_consensus.cc:482] T 08e4d119d90f4146ba53a1b18efe2386 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:10.736344 2605 raft_consensus.cc:504] T 08e4d119d90f4146ba53a1b18efe2386 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:10.737833 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08e4d119d90f4146ba53a1b18efe2386" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
W20240503 02:13:10.738531 32709 leader_election.cc:341] T 08e4d119d90f4146ba53a1b18efe2386 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:10.738412 29730 catalog_manager.cc:5484] T f0766669c2494c7db1590730b2c163ca P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6). New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:10.739576 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08e4d119d90f4146ba53a1b18efe2386" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
W20240503 02:13:10.740332 32709 leader_election.cc:341] T 08e4d119d90f4146ba53a1b18efe2386 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:10.740489 32709 leader_election.cc:302] T 08e4d119d90f4146ba53a1b18efe2386 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead; no voters: 9a4d92bc25374fb3960fde7d10be309d, fbbc2be6382e47908ecb7c68c367bd26
I20240503 02:13:10.740830 2604 raft_consensus.cc:2726] T 08e4d119d90f4146ba53a1b18efe2386 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:10.741966 2605 leader_election.cc:288] T 08e4d119d90f4146ba53a1b18efe2386 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:10.735867 2665 consensus_queue.cc:260] T 7263a253c5884bce987d28e188ccca11 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:10.746757 2665 ts_tablet_manager.cc:1398] T 7263a253c5884bce987d28e188ccca11 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240503 02:13:10.759390 2666 tablet_bootstrap.cc:492] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:10.759590 2666 ts_tablet_manager.cc:1367] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.058s user 0.003s sys 0.007s
I20240503 02:13:10.760280 2666 raft_consensus.cc:348] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:10.760576 2666 raft_consensus.cc:374] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:10.760668 2666 raft_consensus.cc:729] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:10.760929 2666 consensus_queue.cc:260] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:10.761698 2666 ts_tablet_manager.cc:1398] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:13:10.762202 2605 raft_consensus.cc:482] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:10.762362 2605 raft_consensus.cc:504] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:10.763095 2605 leader_election.cc:288] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:13:10.763772 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4d497de05c49edbd6e0edac73c058d" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:10.764029 567 raft_consensus.cc:2454] T ac4d497de05c49edbd6e0edac73c058d P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:13:10.764811 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4d497de05c49edbd6e0edac73c058d" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:13:10.766471 32708 leader_election.cc:302] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:10.767055 2604 raft_consensus.cc:2781] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:10.767179 2604 raft_consensus.cc:482] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:10.767278 2604 raft_consensus.cc:3037] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:13:10.769696 32709 leader_election.cc:341] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:10.772935 2604 raft_consensus.cc:504] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:10.774691 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4d497de05c49edbd6e0edac73c058d" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:13:10.774920 567 raft_consensus.cc:3037] T ac4d497de05c49edbd6e0edac73c058d P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:10.777804 567 raft_consensus.cc:2454] T ac4d497de05c49edbd6e0edac73c058d P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:13:10.778000 2604 leader_election.cc:288] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Requested vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:13:10.778474 32708 leader_election.cc:302] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:10.778764 2375 raft_consensus.cc:2781] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:10.778918 2375 raft_consensus.cc:686] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Becoming Leader. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Running, Role: LEADER
I20240503 02:13:10.778951 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4d497de05c49edbd6e0edac73c058d" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:13:10.779179 2375 consensus_queue.cc:237] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
W20240503 02:13:10.779523 32709 leader_election.cc:341] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:10.779836 2666 tablet_bootstrap.cc:492] T ac4d497de05c49edbd6e0edac73c058d P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:10.791033 2556 tablet_bootstrap.cc:492] T 08e4d119d90f4146ba53a1b18efe2386 P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:10.791254 2556 ts_tablet_manager.cc:1367] T 08e4d119d90f4146ba53a1b18efe2386 P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.070s user 0.018s sys 0.035s
I20240503 02:13:10.792183 2556 raft_consensus.cc:348] T 08e4d119d90f4146ba53a1b18efe2386 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:10.792372 2666 tablet_bootstrap.cc:654] T ac4d497de05c49edbd6e0edac73c058d P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:10.792500 2556 raft_consensus.cc:374] T 08e4d119d90f4146ba53a1b18efe2386 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:10.792577 2556 raft_consensus.cc:729] T 08e4d119d90f4146ba53a1b18efe2386 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:10.792879 2556 consensus_queue.cc:260] T 08e4d119d90f4146ba53a1b18efe2386 P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:10.796497 2556 ts_tablet_manager.cc:1398] T 08e4d119d90f4146ba53a1b18efe2386 P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:13:10.806603 2666 tablet_bootstrap.cc:492] T ac4d497de05c49edbd6e0edac73c058d P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:10.806876 2666 ts_tablet_manager.cc:1367] T ac4d497de05c49edbd6e0edac73c058d P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.045s user 0.001s sys 0.005s
I20240503 02:13:10.808369 2666 raft_consensus.cc:348] T ac4d497de05c49edbd6e0edac73c058d P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:10.808681 2666 raft_consensus.cc:374] T ac4d497de05c49edbd6e0edac73c058d P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:10.808837 2666 raft_consensus.cc:729] T ac4d497de05c49edbd6e0edac73c058d P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:10.809170 2666 consensus_queue.cc:260] T ac4d497de05c49edbd6e0edac73c058d P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:10.811259 2666 ts_tablet_manager.cc:1398] T ac4d497de05c49edbd6e0edac73c058d P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:13:10.847131 2580 raft_consensus.cc:482] T 7263a253c5884bce987d28e188ccca11 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:10.847343 2580 raft_consensus.cc:504] T 7263a253c5884bce987d28e188ccca11 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:10.848033 2580 leader_election.cc:288] T 7263a253c5884bce987d28e188ccca11 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:10.848338 29730 catalog_manager.cc:5484] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6). New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:10.848873 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7263a253c5884bce987d28e188ccca11" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:13:10.849135 30265 raft_consensus.cc:2454] T 7263a253c5884bce987d28e188ccca11 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 0.
I20240503 02:13:10.849131 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7263a253c5884bce987d28e188ccca11" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:13:10.849331 437 raft_consensus.cc:2454] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 0.
I20240503 02:13:10.849927 32574 leader_election.cc:302] T 7263a253c5884bce987d28e188ccca11 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:10.850368 2580 raft_consensus.cc:2781] T 7263a253c5884bce987d28e188ccca11 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:10.850466 2580 raft_consensus.cc:482] T 7263a253c5884bce987d28e188ccca11 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:10.850559 2580 raft_consensus.cc:3037] T 7263a253c5884bce987d28e188ccca11 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:10.853631 2580 raft_consensus.cc:504] T 7263a253c5884bce987d28e188ccca11 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:10.854614 2580 leader_election.cc:288] T 7263a253c5884bce987d28e188ccca11 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 election: Requested vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:10.858526 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7263a253c5884bce987d28e188ccca11" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:13:10.858820 437 raft_consensus.cc:3037] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:10.859272 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7263a253c5884bce987d28e188ccca11" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:13:10.859468 30265 raft_consensus.cc:3037] T 7263a253c5884bce987d28e188ccca11 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:10.861977 437 raft_consensus.cc:2454] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 1.
I20240503 02:13:10.862605 32574 leader_election.cc:302] T 7263a253c5884bce987d28e188ccca11 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:10.862823 30265 raft_consensus.cc:2454] T 7263a253c5884bce987d28e188ccca11 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 1.
I20240503 02:13:10.863158 2580 raft_consensus.cc:2781] T 7263a253c5884bce987d28e188ccca11 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:10.863799 2580 raft_consensus.cc:686] T 7263a253c5884bce987d28e188ccca11 P 9a4d92bc25374fb3960fde7d10be309d [term 1 LEADER]: Becoming Leader. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Running, Role: LEADER
I20240503 02:13:10.864132 2580 consensus_queue.cc:237] T 7263a253c5884bce987d28e188ccca11 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:10.876549 2412 raft_consensus.cc:482] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:10.876720 2412 raft_consensus.cc:504] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:10.877447 2412 leader_election.cc:288] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:13:10.878334 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08e4d119d90f4146ba53a1b18efe2386" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:13:10.878513 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08e4d119d90f4146ba53a1b18efe2386" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:13:10.878681 307 raft_consensus.cc:2454] T 08e4d119d90f4146ba53a1b18efe2386 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 0.
I20240503 02:13:10.878805 32640 raft_consensus.cc:2454] T 08e4d119d90f4146ba53a1b18efe2386 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 0.
I20240503 02:13:10.879455 371 leader_election.cc:302] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:10.881253 2412 raft_consensus.cc:2781] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:10.881373 2412 raft_consensus.cc:482] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:10.881453 2412 raft_consensus.cc:3037] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:10.884472 2412 raft_consensus.cc:504] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:10.886055 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08e4d119d90f4146ba53a1b18efe2386" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:13:10.886332 307 raft_consensus.cc:3037] T 08e4d119d90f4146ba53a1b18efe2386 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:10.891891 29732 catalog_manager.cc:5484] T 7263a253c5884bce987d28e188ccca11 P 9a4d92bc25374fb3960fde7d10be309d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5). New cstate: current_term: 1 leader_uuid: "9a4d92bc25374fb3960fde7d10be309d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:10.908129 307 raft_consensus.cc:2454] T 08e4d119d90f4146ba53a1b18efe2386 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 1.
I20240503 02:13:10.908615 2412 leader_election.cc:288] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Requested vote from peers 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:13:10.939639 29730 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:13:10.964004 567 raft_consensus.cc:1261] T f0766669c2494c7db1590730b2c163ca P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:10.964849 2375 consensus_queue.cc:1035] T f0766669c2494c7db1590730b2c163ca P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:10.973899 32640 raft_consensus.cc:1261] T f0766669c2494c7db1590730b2c163ca P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:10.974665 2375 consensus_queue.cc:1035] T f0766669c2494c7db1590730b2c163ca P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:10.982228 29730 catalog_manager.cc:5576] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595) does not have the latest schema for tablet f0766669c2494c7db1590730b2c163ca (table rebalancer_dynamic_table [id=9c822d1fd0454726a711868b2428a47c]). Expected version 1 got 0
I20240503 02:13:10.985822 2702 tablet.cc:1722] T f0766669c2494c7db1590730b2c163ca P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:10.986403 2702 tablet.cc:1620] T f0766669c2494c7db1590730b2c163ca P 8a666e798fe8477d8dd1e51b385d5ead: MemRowSet was empty: no flush needed.
W20240503 02:13:10.995678 371 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.5:35595, user_credentials={real_user=slave}} blocked reactor thread for 103976us
I20240503 02:13:10.996992 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08e4d119d90f4146ba53a1b18efe2386" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d"
I20240503 02:13:10.997332 32641 raft_consensus.cc:3037] T 08e4d119d90f4146ba53a1b18efe2386 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:10.998304 2703 tablet.cc:1722] T f0766669c2494c7db1590730b2c163ca P 9a4d92bc25374fb3960fde7d10be309d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:10.998889 29730 catalog_manager.cc:5576] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595) does not have the latest schema for tablet f0766669c2494c7db1590730b2c163ca (table rebalancer_dynamic_table [id=9c822d1fd0454726a711868b2428a47c]). Expected version 1 got 0
I20240503 02:13:10.999138 2703 tablet.cc:1620] T f0766669c2494c7db1590730b2c163ca P 9a4d92bc25374fb3960fde7d10be309d: MemRowSet was empty: no flush needed.
I20240503 02:13:11.000317 32641 raft_consensus.cc:2454] T 08e4d119d90f4146ba53a1b18efe2386 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 1.
I20240503 02:13:11.008462 371 leader_election.cc:302] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:11.010097 29725 catalog_manager.cc:5576] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233) does not have the latest schema for tablet f0766669c2494c7db1590730b2c163ca (table rebalancer_dynamic_table [id=9c822d1fd0454726a711868b2428a47c]). Expected version 1 got 0
I20240503 02:13:11.015424 29730 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=9c822d1fd0454726a711868b2428a47c] alter complete (version 1)
I20240503 02:13:11.025897 1900 raft_consensus.cc:2781] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:11.043071 1900 raft_consensus.cc:686] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Becoming Leader. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Running, Role: LEADER
I20240503 02:13:11.045680 1900 consensus_queue.cc:237] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:11.059603 2711 tablet.cc:1722] T f0766669c2494c7db1590730b2c163ca P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:11.060226 2711 tablet.cc:1620] T f0766669c2494c7db1590730b2c163ca P f575ce92c0224eeb80b9fae5f6561fbe: MemRowSet was empty: no flush needed.
I20240503 02:13:11.066823 29731 catalog_manager.cc:5576] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233) does not have the latest schema for tablet f0766669c2494c7db1590730b2c163ca (table rebalancer_dynamic_table [id=9c822d1fd0454726a711868b2428a47c]). Expected version 1 got 0
I20240503 02:13:11.060961 29730 catalog_manager.cc:5484] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7). New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:11.124745 29725 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:13:11.142089 2711 tablet.cc:1722] T f0766669c2494c7db1590730b2c163ca P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:11.142650 2711 tablet.cc:1620] T f0766669c2494c7db1590730b2c163ca P f575ce92c0224eeb80b9fae5f6561fbe: MemRowSet was empty: no flush needed.
I20240503 02:13:11.142616 2702 tablet.cc:1722] T f0766669c2494c7db1590730b2c163ca P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:11.143227 2702 tablet.cc:1620] T f0766669c2494c7db1590730b2c163ca P 8a666e798fe8477d8dd1e51b385d5ead: MemRowSet was empty: no flush needed.
I20240503 02:13:11.157650 29730 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=9c822d1fd0454726a711868b2428a47c] alter complete (version 2)
I20240503 02:13:11.162480 2703 tablet.cc:1722] T f0766669c2494c7db1590730b2c163ca P 9a4d92bc25374fb3960fde7d10be309d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:11.163406 2703 tablet.cc:1620] T f0766669c2494c7db1590730b2c163ca P 9a4d92bc25374fb3960fde7d10be309d: MemRowSet was empty: no flush needed.
I20240503 02:13:11.183856 29984 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:13:11.210368 2605 consensus_queue.cc:1035] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:11.218345 437 raft_consensus.cc:3037] T ac4d497de05c49edbd6e0edac73c058d P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:11.219903 29853 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:11.221906 2604 consensus_queue.cc:1035] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:11.256253 1787 consensus_queue.cc:1035] T 7263a253c5884bce987d28e188ccca11 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:11.260722 29732 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:13:11.274475 2702 tablet.cc:1722] T f0766669c2494c7db1590730b2c163ca P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:11.280301 2703 tablet.cc:1722] T f0766669c2494c7db1590730b2c163ca P 9a4d92bc25374fb3960fde7d10be309d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:11.290383 2711 tablet.cc:1722] T f0766669c2494c7db1590730b2c163ca P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:11.300765 29732 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table_new_name [id=9c822d1fd0454726a711868b2428a47c] alter complete (version 3)
I20240503 02:13:11.306025 30245 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:11.328671 2580 consensus_queue.cc:1035] T 7263a253c5884bce987d28e188ccca11 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:11.344558 32755 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:11.362771 30116 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:11.388092 32619 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:11.407639 29732 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:11.407903 29732 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:11.413116 29732 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet f0766669c2494c7db1590730b2c163ca
I20240503 02:13:11.414176 32619 tablet_service.cc:1514] Processing DeleteTablet for tablet f0766669c2494c7db1590730b2c163ca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:11 UTC) from {username='slave'} at 127.0.0.1:38200
I20240503 02:13:11.415227 32755 tablet_service.cc:1514] Processing DeleteTablet for tablet f0766669c2494c7db1590730b2c163ca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:11 UTC) from {username='slave'} at 127.0.0.1:46360
I20240503 02:13:11.423230 548 tablet_service.cc:1514] Processing DeleteTablet for tablet f0766669c2494c7db1590730b2c163ca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:11 UTC) from {username='slave'} at 127.0.0.1:39028
I20240503 02:13:11.429065 2742 tablet_replica.cc:330] T f0766669c2494c7db1590730b2c163ca P 8a666e798fe8477d8dd1e51b385d5ead: stopping tablet replica
I20240503 02:13:11.429782 2742 raft_consensus.cc:2229] T f0766669c2494c7db1590730b2c163ca P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:11.434300 2743 tablet_replica.cc:330] T f0766669c2494c7db1590730b2c163ca P f575ce92c0224eeb80b9fae5f6561fbe: stopping tablet replica
I20240503 02:13:11.434651 2743 raft_consensus.cc:2229] T f0766669c2494c7db1590730b2c163ca P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:11.434962 2743 raft_consensus.cc:2258] T f0766669c2494c7db1590730b2c163ca P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:11.436386 2743 ts_tablet_manager.cc:1875] T f0766669c2494c7db1590730b2c163ca P f575ce92c0224eeb80b9fae5f6561fbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:11.436607 2742 raft_consensus.cc:2258] T f0766669c2494c7db1590730b2c163ca P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:11.440181 2742 ts_tablet_manager.cc:1875] T f0766669c2494c7db1590730b2c163ca P 8a666e798fe8477d8dd1e51b385d5ead: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:11.454842 416 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:13:11.462906 548 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:11.477331 2742 ts_tablet_manager.cc:1888] T f0766669c2494c7db1590730b2c163ca P 8a666e798fe8477d8dd1e51b385d5ead: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:11.476716 2743 ts_tablet_manager.cc:1888] T f0766669c2494c7db1590730b2c163ca P f575ce92c0224eeb80b9fae5f6561fbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:11.478281 2742 log.cc:1196] T f0766669c2494c7db1590730b2c163ca P 8a666e798fe8477d8dd1e51b385d5ead: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal/wals/f0766669c2494c7db1590730b2c163ca
I20240503 02:13:11.478557 2743 log.cc:1196] T f0766669c2494c7db1590730b2c163ca P f575ce92c0224eeb80b9fae5f6561fbe: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal/wals/f0766669c2494c7db1590730b2c163ca
I20240503 02:13:11.478883 2742 ts_tablet_manager.cc:1909] T f0766669c2494c7db1590730b2c163ca P 8a666e798fe8477d8dd1e51b385d5ead: Deleting consensus metadata
I20240503 02:13:11.479416 2743 ts_tablet_manager.cc:1909] T f0766669c2494c7db1590730b2c163ca P f575ce92c0224eeb80b9fae5f6561fbe: Deleting consensus metadata
I20240503 02:13:11.483858 29719 catalog_manager.cc:4830] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): tablet f0766669c2494c7db1590730b2c163ca (table rebalancer_dynamic_table_new_name [id=9c822d1fd0454726a711868b2428a47c]) successfully deleted
I20240503 02:13:11.500957 29718 catalog_manager.cc:4830] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): tablet f0766669c2494c7db1590730b2c163ca (table rebalancer_dynamic_table_new_name [id=9c822d1fd0454726a711868b2428a47c]) successfully deleted
I20240503 02:13:11.516470 29732 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:13:11.526322 29853 tablet_service.cc:1467] Processing CreateTablet for tablet b37d17d280ff477ab0ed86bfca76761f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9893190acca84a29bae558e6e5b93a0d]), partition=
I20240503 02:13:11.526765 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b37d17d280ff477ab0ed86bfca76761f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.526940 32755 tablet_service.cc:1467] Processing CreateTablet for tablet b37d17d280ff477ab0ed86bfca76761f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9893190acca84a29bae558e6e5b93a0d]), partition=
I20240503 02:13:11.527415 32755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b37d17d280ff477ab0ed86bfca76761f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.533643 416 tablet_service.cc:1467] Processing CreateTablet for tablet b37d17d280ff477ab0ed86bfca76761f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9893190acca84a29bae558e6e5b93a0d]), partition=
I20240503 02:13:11.534029 416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b37d17d280ff477ab0ed86bfca76761f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.554062 2708 consensus_queue.cc:1035] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:11.557127 2750 tablet_bootstrap.cc:492] T b37d17d280ff477ab0ed86bfca76761f P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:11.558916 2750 tablet_bootstrap.cc:654] T b37d17d280ff477ab0ed86bfca76761f P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.570618 2694 consensus_queue.cc:1035] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:11.584803 2741 tablet_replica.cc:330] T f0766669c2494c7db1590730b2c163ca P 9a4d92bc25374fb3960fde7d10be309d: stopping tablet replica
I20240503 02:13:11.585640 2747 tablet_bootstrap.cc:492] T b37d17d280ff477ab0ed86bfca76761f P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
W20240503 02:13:11.586969 502 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.5:35595, user_credentials={real_user=slave}} blocked reactor thread for 89168us
I20240503 02:13:11.597398 2741 raft_consensus.cc:2229] T f0766669c2494c7db1590730b2c163ca P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:11.597846 2741 raft_consensus.cc:2258] T f0766669c2494c7db1590730b2c163ca P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:11.599220 2741 ts_tablet_manager.cc:1875] T f0766669c2494c7db1590730b2c163ca P 9a4d92bc25374fb3960fde7d10be309d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:11.604279 2748 tablet_bootstrap.cc:492] T b37d17d280ff477ab0ed86bfca76761f P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:13:11.606985 2748 tablet_bootstrap.cc:654] T b37d17d280ff477ab0ed86bfca76761f P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.608979 2747 tablet_bootstrap.cc:654] T b37d17d280ff477ab0ed86bfca76761f P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.619143 2748 tablet_bootstrap.cc:492] T b37d17d280ff477ab0ed86bfca76761f P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:13:11.619386 2748 ts_tablet_manager.cc:1367] T b37d17d280ff477ab0ed86bfca76761f P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.065s user 0.007s sys 0.004s
I20240503 02:13:11.620409 2748 raft_consensus.cc:348] T b37d17d280ff477ab0ed86bfca76761f P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:11.620823 2748 raft_consensus.cc:374] T b37d17d280ff477ab0ed86bfca76761f P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.621110 2748 raft_consensus.cc:729] T b37d17d280ff477ab0ed86bfca76761f P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.621588 2748 consensus_queue.cc:260] T b37d17d280ff477ab0ed86bfca76761f P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:11.629788 2748 ts_tablet_manager.cc:1398] T b37d17d280ff477ab0ed86bfca76761f P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 02:13:11.632377 2750 tablet_bootstrap.cc:492] T b37d17d280ff477ab0ed86bfca76761f P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:11.632614 2750 ts_tablet_manager.cc:1367] T b37d17d280ff477ab0ed86bfca76761f P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.076s user 0.033s sys 0.023s
I20240503 02:13:11.634011 2750 raft_consensus.cc:348] T b37d17d280ff477ab0ed86bfca76761f P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:11.634307 2750 raft_consensus.cc:374] T b37d17d280ff477ab0ed86bfca76761f P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.634442 2750 raft_consensus.cc:729] T b37d17d280ff477ab0ed86bfca76761f P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.644098 2747 tablet_bootstrap.cc:492] T b37d17d280ff477ab0ed86bfca76761f P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:11.634754 2750 consensus_queue.cc:260] T b37d17d280ff477ab0ed86bfca76761f P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:11.644359 2747 ts_tablet_manager.cc:1367] T b37d17d280ff477ab0ed86bfca76761f P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.059s user 0.005s sys 0.000s
I20240503 02:13:11.645814 2747 raft_consensus.cc:348] T b37d17d280ff477ab0ed86bfca76761f P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:11.646958 2741 ts_tablet_manager.cc:1888] T f0766669c2494c7db1590730b2c163ca P 9a4d92bc25374fb3960fde7d10be309d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:11.647792 2747 raft_consensus.cc:374] T b37d17d280ff477ab0ed86bfca76761f P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.647985 2747 raft_consensus.cc:729] T b37d17d280ff477ab0ed86bfca76761f P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.648348 2747 consensus_queue.cc:260] T b37d17d280ff477ab0ed86bfca76761f P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:11.649943 2744 raft_consensus.cc:482] T b37d17d280ff477ab0ed86bfca76761f P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:11.650101 2744 raft_consensus.cc:504] T b37d17d280ff477ab0ed86bfca76761f P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:11.650441 2747 ts_tablet_manager.cc:1398] T b37d17d280ff477ab0ed86bfca76761f P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240503 02:13:11.650923 2741 log.cc:1196] T f0766669c2494c7db1590730b2c163ca P 9a4d92bc25374fb3960fde7d10be309d: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal/wals/f0766669c2494c7db1590730b2c163ca
I20240503 02:13:11.650985 2744 leader_election.cc:288] T b37d17d280ff477ab0ed86bfca76761f P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:11.651520 2741 ts_tablet_manager.cc:1909] T f0766669c2494c7db1590730b2c163ca P 9a4d92bc25374fb3960fde7d10be309d: Deleting consensus metadata
I20240503 02:13:11.653527 435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b37d17d280ff477ab0ed86bfca76761f" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:13:11.653752 435 raft_consensus.cc:2454] T b37d17d280ff477ab0ed86bfca76761f P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:13:11.654194 29809 leader_election.cc:302] T b37d17d280ff477ab0ed86bfca76761f P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:11.654729 1484 raft_consensus.cc:2781] T b37d17d280ff477ab0ed86bfca76761f P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:11.654863 1484 raft_consensus.cc:482] T b37d17d280ff477ab0ed86bfca76761f P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:11.654824 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b37d17d280ff477ab0ed86bfca76761f" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:13:11.654964 1484 raft_consensus.cc:3037] T b37d17d280ff477ab0ed86bfca76761f P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:11.655064 307 raft_consensus.cc:2454] T b37d17d280ff477ab0ed86bfca76761f P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:13:11.657770 1484 raft_consensus.cc:504] T b37d17d280ff477ab0ed86bfca76761f P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:11.658389 1484 leader_election.cc:288] T b37d17d280ff477ab0ed86bfca76761f P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Requested vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:11.660116 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b37d17d280ff477ab0ed86bfca76761f" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:13:11.660298 307 raft_consensus.cc:3037] T b37d17d280ff477ab0ed86bfca76761f P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:11.660601 2750 ts_tablet_manager.cc:1398] T b37d17d280ff477ab0ed86bfca76761f P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.024s user 0.000s sys 0.002s
I20240503 02:13:11.662468 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b37d17d280ff477ab0ed86bfca76761f" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:13:11.662940 29725 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_extra_table_30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:11.664376 437 raft_consensus.cc:3037] T b37d17d280ff477ab0ed86bfca76761f P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:11.665304 307 raft_consensus.cc:2454] T b37d17d280ff477ab0ed86bfca76761f P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:13:11.665992 29809 leader_election.cc:302] T b37d17d280ff477ab0ed86bfca76761f P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, 8a666e798fe8477d8dd1e51b385d5ead; no voters:
I20240503 02:13:11.666253 2744 raft_consensus.cc:2781] T b37d17d280ff477ab0ed86bfca76761f P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:11.666407 2744 raft_consensus.cc:686] T b37d17d280ff477ab0ed86bfca76761f P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Becoming Leader. State: Replica: 16440e365430405fb10366ab70e2b429, State: Running, Role: LEADER
I20240503 02:13:11.666676 2744 consensus_queue.cc:237] T b37d17d280ff477ab0ed86bfca76761f P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:11.667428 437 raft_consensus.cc:2454] T b37d17d280ff477ab0ed86bfca76761f P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:13:11.667591 29719 catalog_manager.cc:4830] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): tablet f0766669c2494c7db1590730b2c163ca (table rebalancer_dynamic_table_new_name [id=9c822d1fd0454726a711868b2428a47c]) successfully deleted
I20240503 02:13:11.675521 548 tablet_service.cc:1467] Processing CreateTablet for tablet ae4373a66af14c8f8a633963dffd2261 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=854d24d23fe34d699e6656ef51d15693]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.676122 548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae4373a66af14c8f8a633963dffd2261. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.677666 32619 tablet_service.cc:1467] Processing CreateTablet for tablet ae4373a66af14c8f8a633963dffd2261 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=854d24d23fe34d699e6656ef51d15693]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.677623 416 tablet_service.cc:1467] Processing CreateTablet for tablet ae4373a66af14c8f8a633963dffd2261 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=854d24d23fe34d699e6656ef51d15693]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.678071 416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae4373a66af14c8f8a633963dffd2261. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.678066 32619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae4373a66af14c8f8a633963dffd2261. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.679248 547 tablet_service.cc:1467] Processing CreateTablet for tablet fb8fcfa3f3ab46b182b692a0c4fc61ed (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=854d24d23fe34d699e6656ef51d15693]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.679705 547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb8fcfa3f3ab46b182b692a0c4fc61ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.681419 32620 tablet_service.cc:1467] Processing CreateTablet for tablet fb8fcfa3f3ab46b182b692a0c4fc61ed (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=854d24d23fe34d699e6656ef51d15693]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.681833 32620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb8fcfa3f3ab46b182b692a0c4fc61ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.684759 546 tablet_service.cc:1467] Processing CreateTablet for tablet b76b807d2eb24f3ead50bdc0bb1b0231 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=854d24d23fe34d699e6656ef51d15693]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.685226 546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b76b807d2eb24f3ead50bdc0bb1b0231. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.690090 30116 tablet_service.cc:1467] Processing CreateTablet for tablet b76b807d2eb24f3ead50bdc0bb1b0231 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=854d24d23fe34d699e6656ef51d15693]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.690552 30116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b76b807d2eb24f3ead50bdc0bb1b0231. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.693897 30244 tablet_service.cc:1467] Processing CreateTablet for tablet b76b807d2eb24f3ead50bdc0bb1b0231 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=854d24d23fe34d699e6656ef51d15693]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.693933 30245 tablet_service.cc:1467] Processing CreateTablet for tablet fb8fcfa3f3ab46b182b692a0c4fc61ed (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=854d24d23fe34d699e6656ef51d15693]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.694344 30244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b76b807d2eb24f3ead50bdc0bb1b0231. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.696678 30245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb8fcfa3f3ab46b182b692a0c4fc61ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.697016 2750 tablet_bootstrap.cc:492] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:11.698864 2750 tablet_bootstrap.cc:654] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.708876 2760 tablet_bootstrap.cc:492] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:13:11.713459 2760 tablet_bootstrap.cc:654] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.716625 2758 tablet_bootstrap.cc:492] T b76b807d2eb24f3ead50bdc0bb1b0231 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:11.719483 2750 tablet_bootstrap.cc:492] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:11.719695 2750 ts_tablet_manager.cc:1367] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.005s
I20240503 02:13:11.720769 2750 raft_consensus.cc:348] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:11.721050 2750 raft_consensus.cc:374] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.721221 2750 raft_consensus.cc:729] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.721640 2750 consensus_queue.cc:260] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:11.722287 2750 ts_tablet_manager.cc:1398] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:13:11.733894 2758 tablet_bootstrap.cc:654] T b76b807d2eb24f3ead50bdc0bb1b0231 P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.739529 2760 tablet_bootstrap.cc:492] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:13:11.739734 2760 ts_tablet_manager.cc:1367] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.000s
I20240503 02:13:11.740584 2760 raft_consensus.cc:348] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:11.740800 2760 raft_consensus.cc:374] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.740883 2760 raft_consensus.cc:729] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.742666 29732 catalog_manager.cc:5484] T b37d17d280ff477ab0ed86bfca76761f P 16440e365430405fb10366ab70e2b429 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16440e365430405fb10366ab70e2b429 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:11.743455 2758 tablet_bootstrap.cc:492] T b76b807d2eb24f3ead50bdc0bb1b0231 P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:11.746920 2758 ts_tablet_manager.cc:1367] T b76b807d2eb24f3ead50bdc0bb1b0231 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.000s
I20240503 02:13:11.747807 2758 raft_consensus.cc:348] T b76b807d2eb24f3ead50bdc0bb1b0231 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:11.748087 2758 raft_consensus.cc:374] T b76b807d2eb24f3ead50bdc0bb1b0231 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.748205 2758 raft_consensus.cc:729] T b76b807d2eb24f3ead50bdc0bb1b0231 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.748450 2760 consensus_queue.cc:260] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:11.748502 2758 consensus_queue.cc:260] T b76b807d2eb24f3ead50bdc0bb1b0231 P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:11.750416 2758 ts_tablet_manager.cc:1398] T b76b807d2eb24f3ead50bdc0bb1b0231 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:13:11.750766 2758 tablet_bootstrap.cc:492] T ae4373a66af14c8f8a633963dffd2261 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:11.751152 2760 ts_tablet_manager.cc:1398] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 02:13:11.754889 2758 tablet_bootstrap.cc:654] T ae4373a66af14c8f8a633963dffd2261 P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.763635 2728 raft_consensus.cc:482] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:11.763828 2728 raft_consensus.cc:504] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:11.764493 2728 leader_election.cc:288] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
W20240503 02:13:11.766115 30199 leader_election.cc:341] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Not found: Tablet not found: b76b807d2eb24f3ead50bdc0bb1b0231
I20240503 02:13:11.766444 566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76b807d2eb24f3ead50bdc0bb1b0231" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:11.766676 566 raft_consensus.cc:2454] T b76b807d2eb24f3ead50bdc0bb1b0231 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:13:11.767099 30199 leader_election.cc:302] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, f575ce92c0224eeb80b9fae5f6561fbe; no voters: a9094240860e43bfb129193f3bfd5be6
I20240503 02:13:11.767354 2728 raft_consensus.cc:2781] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:11.767457 2728 raft_consensus.cc:482] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:11.767547 2728 raft_consensus.cc:3037] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:11.767658 2758 tablet_bootstrap.cc:492] T ae4373a66af14c8f8a633963dffd2261 P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:11.767864 2758 ts_tablet_manager.cc:1367] T ae4373a66af14c8f8a633963dffd2261 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240503 02:13:11.768901 2758 raft_consensus.cc:348] T ae4373a66af14c8f8a633963dffd2261 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:11.769207 2758 raft_consensus.cc:374] T ae4373a66af14c8f8a633963dffd2261 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.769290 2758 raft_consensus.cc:729] T ae4373a66af14c8f8a633963dffd2261 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.769698 2758 consensus_queue.cc:260] T ae4373a66af14c8f8a633963dffd2261 P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:11.770429 2728 raft_consensus.cc:504] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:11.772580 2728 leader_election.cc:288] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Requested vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
W20240503 02:13:11.774993 30199 leader_election.cc:341] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Not found: Tablet not found: b76b807d2eb24f3ead50bdc0bb1b0231
I20240503 02:13:11.775279 566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76b807d2eb24f3ead50bdc0bb1b0231" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:13:11.775415 2758 ts_tablet_manager.cc:1398] T ae4373a66af14c8f8a633963dffd2261 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240503 02:13:11.775604 566 raft_consensus.cc:3037] T b76b807d2eb24f3ead50bdc0bb1b0231 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:11.789940 2760 tablet_bootstrap.cc:492] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:13:11.790482 566 raft_consensus.cc:2454] T b76b807d2eb24f3ead50bdc0bb1b0231 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:13:11.790964 30199 leader_election.cc:302] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, f575ce92c0224eeb80b9fae5f6561fbe; no voters: a9094240860e43bfb129193f3bfd5be6
I20240503 02:13:11.791301 2728 raft_consensus.cc:2781] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:11.791427 2728 raft_consensus.cc:686] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Becoming Leader. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Running, Role: LEADER
I20240503 02:13:11.791759 2760 tablet_bootstrap.cc:654] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.791728 2728 consensus_queue.cc:237] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:11.794620 2760 tablet_bootstrap.cc:492] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:13:11.796450 2760 ts_tablet_manager.cc:1367] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:13:11.797513 2760 raft_consensus.cc:348] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:11.797850 2760 raft_consensus.cc:374] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.797942 2760 raft_consensus.cc:729] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.798365 2758 tablet_bootstrap.cc:492] T fb8fcfa3f3ab46b182b692a0c4fc61ed P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:11.798264 2760 consensus_queue.cc:260] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:11.799043 2760 ts_tablet_manager.cc:1398] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:11.800160 2758 tablet_bootstrap.cc:654] T fb8fcfa3f3ab46b182b692a0c4fc61ed P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.814224 2758 tablet_bootstrap.cc:492] T fb8fcfa3f3ab46b182b692a0c4fc61ed P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:11.814463 2758 ts_tablet_manager.cc:1367] T fb8fcfa3f3ab46b182b692a0c4fc61ed P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.002s
I20240503 02:13:11.814872 29725 catalog_manager.cc:5484] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4). New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:11.816907 2758 raft_consensus.cc:348] T fb8fcfa3f3ab46b182b692a0c4fc61ed P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:11.817189 2758 raft_consensus.cc:374] T fb8fcfa3f3ab46b182b692a0c4fc61ed P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.817305 2758 raft_consensus.cc:729] T fb8fcfa3f3ab46b182b692a0c4fc61ed P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.817639 2758 consensus_queue.cc:260] T fb8fcfa3f3ab46b182b692a0c4fc61ed P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:11.818213 2758 ts_tablet_manager.cc:1398] T fb8fcfa3f3ab46b182b692a0c4fc61ed P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:13:11.838774 2757 tablet_bootstrap.cc:492] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:11.841570 2767 tablet_bootstrap.cc:492] T b76b807d2eb24f3ead50bdc0bb1b0231 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:13:11.843292 2757 tablet_bootstrap.cc:654] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.843704 2767 tablet_bootstrap.cc:654] T b76b807d2eb24f3ead50bdc0bb1b0231 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.859939 29732 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:13:11.869971 2757 tablet_bootstrap.cc:492] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:11.870245 2757 ts_tablet_manager.cc:1367] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.000s
I20240503 02:13:11.871171 2757 raft_consensus.cc:348] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:11.871465 2757 raft_consensus.cc:374] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.871593 2757 raft_consensus.cc:729] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.873792 2757 consensus_queue.cc:260] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:11.874606 2757 ts_tablet_manager.cc:1398] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:13:11.876094 2757 tablet_bootstrap.cc:492] T ae4373a66af14c8f8a633963dffd2261 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:11.876365 2767 tablet_bootstrap.cc:492] T b76b807d2eb24f3ead50bdc0bb1b0231 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:13:11.876557 2767 ts_tablet_manager.cc:1367] T b76b807d2eb24f3ead50bdc0bb1b0231 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.035s user 0.010s sys 0.017s
I20240503 02:13:11.877400 2767 raft_consensus.cc:348] T b76b807d2eb24f3ead50bdc0bb1b0231 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:11.877636 2767 raft_consensus.cc:374] T b76b807d2eb24f3ead50bdc0bb1b0231 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.877755 2767 raft_consensus.cc:729] T b76b807d2eb24f3ead50bdc0bb1b0231 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.877995 2767 consensus_queue.cc:260] T b76b807d2eb24f3ead50bdc0bb1b0231 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:11.880328 2757 tablet_bootstrap.cc:654] T ae4373a66af14c8f8a633963dffd2261 P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.932549 307 raft_consensus.cc:1261] T b37d17d280ff477ab0ed86bfca76761f P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:11.933207 2755 consensus_queue.cc:1035] T b37d17d280ff477ab0ed86bfca76761f P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:13:11.942312 30072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.1:36719, user_credentials={real_user=slave}} blocked reactor thread for 76481.4us
I20240503 02:13:11.947377 435 raft_consensus.cc:1261] T b37d17d280ff477ab0ed86bfca76761f P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:11.948132 2755 consensus_queue.cc:1035] T b37d17d280ff477ab0ed86bfca76761f P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:11.956719 29730 catalog_manager.cc:5576] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241) does not have the latest schema for tablet b37d17d280ff477ab0ed86bfca76761f (table rebalancer_dynamic_table [id=9893190acca84a29bae558e6e5b93a0d]). Expected version 1 got 0
I20240503 02:13:11.970242 2778 tablet.cc:1722] T b37d17d280ff477ab0ed86bfca76761f P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:11.970919 2778 tablet.cc:1620] T b37d17d280ff477ab0ed86bfca76761f P fbbc2be6382e47908ecb7c68c367bd26: MemRowSet was empty: no flush needed.
I20240503 02:13:11.972684 2777 tablet.cc:1722] T b37d17d280ff477ab0ed86bfca76761f P 16440e365430405fb10366ab70e2b429: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:11.976173 2777 tablet.cc:1620] T b37d17d280ff477ab0ed86bfca76761f P 16440e365430405fb10366ab70e2b429: MemRowSet was empty: no flush needed.
I20240503 02:13:11.979167 29730 catalog_manager.cc:5576] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241) does not have the latest schema for tablet b37d17d280ff477ab0ed86bfca76761f (table rebalancer_dynamic_table [id=9893190acca84a29bae558e6e5b93a0d]). Expected version 1 got 0
I20240503 02:13:11.992060 2745 raft_consensus.cc:482] T ae4373a66af14c8f8a633963dffd2261 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:11.992261 2745 raft_consensus.cc:504] T ae4373a66af14c8f8a633963dffd2261 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:11.993257 2745 leader_election.cc:288] T ae4373a66af14c8f8a633963dffd2261 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:11.993921 2780 tablet.cc:1722] T b37d17d280ff477ab0ed86bfca76761f P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:11.994271 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4373a66af14c8f8a633963dffd2261" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:13:11.994452 2780 tablet.cc:1620] T b37d17d280ff477ab0ed86bfca76761f P 8a666e798fe8477d8dd1e51b385d5ead: MemRowSet was empty: no flush needed.
I20240503 02:13:11.994676 29730 catalog_manager.cc:5576] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975) does not have the latest schema for tablet b37d17d280ff477ab0ed86bfca76761f (table rebalancer_dynamic_table [id=9893190acca84a29bae558e6e5b93a0d]). Expected version 1 got 0
W20240503 02:13:11.994829 502 leader_election.cc:341] T ae4373a66af14c8f8a633963dffd2261 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:11.995121 435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4373a66af14c8f8a633963dffd2261" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:13:11.995364 435 raft_consensus.cc:2454] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 0.
I20240503 02:13:11.996336 502 leader_election.cc:302] T ae4373a66af14c8f8a633963dffd2261 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f575ce92c0224eeb80b9fae5f6561fbe, fbbc2be6382e47908ecb7c68c367bd26; no voters: 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:13:11.996742 2745 raft_consensus.cc:2781] T ae4373a66af14c8f8a633963dffd2261 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:11.996850 2745 raft_consensus.cc:482] T ae4373a66af14c8f8a633963dffd2261 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:11.996945 2745 raft_consensus.cc:3037] T ae4373a66af14c8f8a633963dffd2261 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:12.000347 2745 raft_consensus.cc:504] T ae4373a66af14c8f8a633963dffd2261 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:12.001904 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4373a66af14c8f8a633963dffd2261" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d"
I20240503 02:13:12.002282 29732 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=9893190acca84a29bae558e6e5b93a0d] alter complete (version 1)
W20240503 02:13:12.002858 502 leader_election.cc:341] T ae4373a66af14c8f8a633963dffd2261 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:12.003710 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4373a66af14c8f8a633963dffd2261" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:13:12.003993 437 raft_consensus.cc:3037] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:12.007444 437 raft_consensus.cc:2454] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 1.
I20240503 02:13:12.009455 502 leader_election.cc:302] T ae4373a66af14c8f8a633963dffd2261 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f575ce92c0224eeb80b9fae5f6561fbe, fbbc2be6382e47908ecb7c68c367bd26; no voters: 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:13:12.009896 2494 raft_consensus.cc:2781] T ae4373a66af14c8f8a633963dffd2261 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:12.010753 2494 raft_consensus.cc:686] T ae4373a66af14c8f8a633963dffd2261 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 LEADER]: Becoming Leader. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Running, Role: LEADER
I20240503 02:13:12.011051 2494 consensus_queue.cc:237] T ae4373a66af14c8f8a633963dffd2261 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:12.016038 2745 leader_election.cc:288] T ae4373a66af14c8f8a633963dffd2261 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 election: Requested vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:12.019093 29732 catalog_manager.cc:5484] T ae4373a66af14c8f8a633963dffd2261 P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: term changed from 0 to 1, leader changed from <none> to f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8). New cstate: current_term: 1 leader_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:12.029196 2757 tablet_bootstrap.cc:492] T ae4373a66af14c8f8a633963dffd2261 P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:12.029440 2757 ts_tablet_manager.cc:1367] T ae4373a66af14c8f8a633963dffd2261 P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.153s user 0.005s sys 0.024s
I20240503 02:13:12.030225 2757 raft_consensus.cc:348] T ae4373a66af14c8f8a633963dffd2261 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:12.031037 2757 raft_consensus.cc:374] T ae4373a66af14c8f8a633963dffd2261 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:12.031220 2757 raft_consensus.cc:729] T ae4373a66af14c8f8a633963dffd2261 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:12.031514 2757 consensus_queue.cc:260] T ae4373a66af14c8f8a633963dffd2261 P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:12.032308 2757 ts_tablet_manager.cc:1398] T ae4373a66af14c8f8a633963dffd2261 P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:12.042768 2767 ts_tablet_manager.cc:1398] T b76b807d2eb24f3ead50bdc0bb1b0231 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.166s user 0.033s sys 0.058s
I20240503 02:13:12.114410 29730 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:13:12.125356 2777 tablet.cc:1722] T b37d17d280ff477ab0ed86bfca76761f P 16440e365430405fb10366ab70e2b429: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:12.125857 2777 tablet.cc:1620] T b37d17d280ff477ab0ed86bfca76761f P 16440e365430405fb10366ab70e2b429: MemRowSet was empty: no flush needed.
I20240503 02:13:12.127032 2778 tablet.cc:1722] T b37d17d280ff477ab0ed86bfca76761f P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:12.127543 2778 tablet.cc:1620] T b37d17d280ff477ab0ed86bfca76761f P fbbc2be6382e47908ecb7c68c367bd26: MemRowSet was empty: no flush needed.
I20240503 02:13:12.137830 29730 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=9893190acca84a29bae558e6e5b93a0d] alter complete (version 2)
I20240503 02:13:12.138413 2780 tablet.cc:1722] T b37d17d280ff477ab0ed86bfca76761f P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:12.138926 2780 tablet.cc:1620] T b37d17d280ff477ab0ed86bfca76761f P 8a666e798fe8477d8dd1e51b385d5ead: MemRowSet was empty: no flush needed.
I20240503 02:13:12.155674 1787 raft_consensus.cc:482] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:12.155866 1787 raft_consensus.cc:504] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:12.156626 1787 leader_election.cc:288] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:13:12.157829 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8fcfa3f3ab46b182b692a0c4fc61ed" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:13:12.158028 30266 raft_consensus.cc:2454] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 0.
I20240503 02:13:12.158617 566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8fcfa3f3ab46b182b692a0c4fc61ed" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:12.158847 566 raft_consensus.cc:2454] T fb8fcfa3f3ab46b182b692a0c4fc61ed P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 0.
I20240503 02:13:12.159685 32573 leader_election.cc:302] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:12.161288 2794 raft_consensus.cc:2781] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:12.161399 2794 raft_consensus.cc:482] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:12.161511 2794 raft_consensus.cc:3037] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:12.163946 2794 raft_consensus.cc:504] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:12.165127 566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8fcfa3f3ab46b182b692a0c4fc61ed" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:13:12.165334 566 raft_consensus.cc:3037] T fb8fcfa3f3ab46b182b692a0c4fc61ed P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:12.165563 2794 leader_election.cc:288] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 election: Requested vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:13:12.165611 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8fcfa3f3ab46b182b692a0c4fc61ed" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:13:12.165786 30266 raft_consensus.cc:3037] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:12.167618 566 raft_consensus.cc:2454] T fb8fcfa3f3ab46b182b692a0c4fc61ed P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 1.
I20240503 02:13:12.168357 30266 raft_consensus.cc:2454] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 1.
I20240503 02:13:12.168972 32573 leader_election.cc:302] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:12.169744 2794 raft_consensus.cc:2781] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:12.170279 2794 raft_consensus.cc:686] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 9a4d92bc25374fb3960fde7d10be309d [term 1 LEADER]: Becoming Leader. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Running, Role: LEADER
I20240503 02:13:12.170564 2794 consensus_queue.cc:237] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:12.179576 29730 catalog_manager.cc:5484] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 9a4d92bc25374fb3960fde7d10be309d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5). New cstate: current_term: 1 leader_uuid: "9a4d92bc25374fb3960fde7d10be309d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:12.180780 30136 raft_consensus.cc:3037] T b76b807d2eb24f3ead50bdc0bb1b0231 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:12.183645 2728 consensus_queue.cc:1035] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:12.209744 29874 consensus_queue.cc:237] T 5f4b72b138c64ac58e6477cd2b1fb215 P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:12.216760 307 raft_consensus.cc:1261] T 5f4b72b138c64ac58e6477cd2b1fb215 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:12.217741 2744 consensus_queue.cc:1035] T 5f4b72b138c64ac58e6477cd2b1fb215 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:12.220727 2755 raft_consensus.cc:2932] T 5f4b72b138c64ac58e6477cd2b1fb215 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:12.221928 2728 consensus_queue.cc:1035] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:12.228482 29730 catalog_manager.cc:5484] T 5f4b72b138c64ac58e6477cd2b1fb215 P 16440e365430405fb10366ab70e2b429 reported cstate change: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:12.229650 30005 raft_consensus.cc:1261] T 5f4b72b138c64ac58e6477cd2b1fb215 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:12.230202 2744 consensus_queue.cc:1035] T 5f4b72b138c64ac58e6477cd2b1fb215 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:12.230389 307 raft_consensus.cc:2932] T 5f4b72b138c64ac58e6477cd2b1fb215 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:12.233003 30116 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f4b72b138c64ac58e6477cd2b1fb215 with delete_type TABLET_DATA_TOMBSTONED (TS a9094240860e43bfb129193f3bfd5be6 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44812
I20240503 02:13:12.233245 2794 raft_consensus.cc:482] T ae4373a66af14c8f8a633963dffd2261 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:12.233413 2794 raft_consensus.cc:504] T ae4373a66af14c8f8a633963dffd2261 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:12.233767 2809 tablet_replica.cc:330] T 5f4b72b138c64ac58e6477cd2b1fb215 P a9094240860e43bfb129193f3bfd5be6: stopping tablet replica
I20240503 02:13:12.233981 2809 raft_consensus.cc:2229] T 5f4b72b138c64ac58e6477cd2b1fb215 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:12.234344 2809 raft_consensus.cc:2258] T 5f4b72b138c64ac58e6477cd2b1fb215 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:12.234350 2794 leader_election.cc:288] T ae4373a66af14c8f8a633963dffd2261 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:12.234586 565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4373a66af14c8f8a633963dffd2261" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:12.235077 2809 ts_tablet_manager.cc:1875] T 5f4b72b138c64ac58e6477cd2b1fb215 P a9094240860e43bfb129193f3bfd5be6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:12.235224 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4373a66af14c8f8a633963dffd2261" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:13:12.235477 437 raft_consensus.cc:2379] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a4d92bc25374fb3960fde7d10be309d in current term 1: Already voted for candidate f575ce92c0224eeb80b9fae5f6561fbe in this term.
I20240503 02:13:12.235996 32574 leader_election.cc:302] T ae4373a66af14c8f8a633963dffd2261 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d; no voters: f575ce92c0224eeb80b9fae5f6561fbe, fbbc2be6382e47908ecb7c68c367bd26
I20240503 02:13:12.237691 2794 raft_consensus.cc:3037] T ae4373a66af14c8f8a633963dffd2261 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:12.240851 2794 raft_consensus.cc:2726] T ae4373a66af14c8f8a633963dffd2261 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:12.242616 29730 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:13:12.244278 2809 ts_tablet_manager.cc:1888] T 5f4b72b138c64ac58e6477cd2b1fb215 P a9094240860e43bfb129193f3bfd5be6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:13:12.244750 2809 log.cc:1196] T 5f4b72b138c64ac58e6477cd2b1fb215 P a9094240860e43bfb129193f3bfd5be6: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/wals/5f4b72b138c64ac58e6477cd2b1fb215
I20240503 02:13:12.254755 2777 tablet.cc:1722] T b37d17d280ff477ab0ed86bfca76761f P 16440e365430405fb10366ab70e2b429: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:12.255627 30005 raft_consensus.cc:2932] T 5f4b72b138c64ac58e6477cd2b1fb215 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:12.257970 29718 catalog_manager.cc:4830] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): tablet 5f4b72b138c64ac58e6477cd2b1fb215 (table rebalance_test_table_11 [id=9498d0be60bf4564a9127c17a357f7ed]) successfully deleted
I20240503 02:13:12.260195 2778 tablet.cc:1722] T b37d17d280ff477ab0ed86bfca76761f P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:12.264227 29730 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table_new_name [id=9893190acca84a29bae558e6e5b93a0d] alter complete (version 3)
I20240503 02:13:12.267777 2780 tablet.cc:1722] T b37d17d280ff477ab0ed86bfca76761f P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:12.368221 29730 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:12.393069 29730 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:12.396907 29730 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet b37d17d280ff477ab0ed86bfca76761f
I20240503 02:13:12.397783 29853 tablet_service.cc:1514] Processing DeleteTablet for tablet b37d17d280ff477ab0ed86bfca76761f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:12 UTC) from {username='slave'} at 127.0.0.1:58744
I20240503 02:13:12.398151 416 tablet_service.cc:1514] Processing DeleteTablet for tablet b37d17d280ff477ab0ed86bfca76761f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:12 UTC) from {username='slave'} at 127.0.0.1:50814
I20240503 02:13:12.398622 32755 tablet_service.cc:1514] Processing DeleteTablet for tablet b37d17d280ff477ab0ed86bfca76761f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:12 UTC) from {username='slave'} at 127.0.0.1:46360
I20240503 02:13:12.398859 2840 tablet_replica.cc:330] T b37d17d280ff477ab0ed86bfca76761f P fbbc2be6382e47908ecb7c68c367bd26: stopping tablet replica
I20240503 02:13:12.399221 2840 raft_consensus.cc:2229] T b37d17d280ff477ab0ed86bfca76761f P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:12.399595 2840 raft_consensus.cc:2258] T b37d17d280ff477ab0ed86bfca76761f P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:12.400925 2840 ts_tablet_manager.cc:1875] T b37d17d280ff477ab0ed86bfca76761f P fbbc2be6382e47908ecb7c68c367bd26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:12.410864 2840 ts_tablet_manager.cc:1888] T b37d17d280ff477ab0ed86bfca76761f P fbbc2be6382e47908ecb7c68c367bd26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:12.411139 2840 log.cc:1196] T b37d17d280ff477ab0ed86bfca76761f P fbbc2be6382e47908ecb7c68c367bd26: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal/wals/b37d17d280ff477ab0ed86bfca76761f
I20240503 02:13:12.411770 2840 ts_tablet_manager.cc:1909] T b37d17d280ff477ab0ed86bfca76761f P fbbc2be6382e47908ecb7c68c367bd26: Deleting consensus metadata
I20240503 02:13:12.413094 29719 catalog_manager.cc:4830] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): tablet b37d17d280ff477ab0ed86bfca76761f (table rebalancer_dynamic_table_new_name [id=9893190acca84a29bae558e6e5b93a0d]) successfully deleted
I20240503 02:13:12.413957 2841 tablet_replica.cc:330] T b37d17d280ff477ab0ed86bfca76761f P 8a666e798fe8477d8dd1e51b385d5ead: stopping tablet replica
I20240503 02:13:12.421294 2841 raft_consensus.cc:2229] T b37d17d280ff477ab0ed86bfca76761f P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:12.421711 2841 raft_consensus.cc:2258] T b37d17d280ff477ab0ed86bfca76761f P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:12.422789 2841 ts_tablet_manager.cc:1875] T b37d17d280ff477ab0ed86bfca76761f P 8a666e798fe8477d8dd1e51b385d5ead: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:12.424875 2839 tablet_replica.cc:330] T b37d17d280ff477ab0ed86bfca76761f P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:13:12.425403 2839 raft_consensus.cc:2229] T b37d17d280ff477ab0ed86bfca76761f P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:12.425769 2839 raft_consensus.cc:2258] T b37d17d280ff477ab0ed86bfca76761f P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:12.429489 2841 ts_tablet_manager.cc:1888] T b37d17d280ff477ab0ed86bfca76761f P 8a666e798fe8477d8dd1e51b385d5ead: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:12.429615 2841 log.cc:1196] T b37d17d280ff477ab0ed86bfca76761f P 8a666e798fe8477d8dd1e51b385d5ead: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal/wals/b37d17d280ff477ab0ed86bfca76761f
I20240503 02:13:12.430061 2841 ts_tablet_manager.cc:1909] T b37d17d280ff477ab0ed86bfca76761f P 8a666e798fe8477d8dd1e51b385d5ead: Deleting consensus metadata
I20240503 02:13:12.431034 29719 catalog_manager.cc:4830] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): tablet b37d17d280ff477ab0ed86bfca76761f (table rebalancer_dynamic_table_new_name [id=9893190acca84a29bae558e6e5b93a0d]) successfully deleted
I20240503 02:13:12.437906 2839 ts_tablet_manager.cc:1875] T b37d17d280ff477ab0ed86bfca76761f P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:12.445940 2839 ts_tablet_manager.cc:1888] T b37d17d280ff477ab0ed86bfca76761f P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:12.446180 2839 log.cc:1196] T b37d17d280ff477ab0ed86bfca76761f P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/b37d17d280ff477ab0ed86bfca76761f
I20240503 02:13:12.446699 2839 ts_tablet_manager.cc:1909] T b37d17d280ff477ab0ed86bfca76761f P 16440e365430405fb10366ab70e2b429: Deleting consensus metadata
I20240503 02:13:12.447722 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet b37d17d280ff477ab0ed86bfca76761f (table rebalancer_dynamic_table_new_name [id=9893190acca84a29bae558e6e5b93a0d]) successfully deleted
I20240503 02:13:12.499475 29731 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:13:12.507380 30116 tablet_service.cc:1467] Processing CreateTablet for tablet a08467832a3043e8afd1d46dc5b4053d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=92b55a490df34e4695ec552b8fa8dee4]), partition=
I20240503 02:13:12.507622 32755 tablet_service.cc:1467] Processing CreateTablet for tablet a08467832a3043e8afd1d46dc5b4053d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=92b55a490df34e4695ec552b8fa8dee4]), partition=
I20240503 02:13:12.507838 30116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a08467832a3043e8afd1d46dc5b4053d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:12.508000 32755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a08467832a3043e8afd1d46dc5b4053d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:12.513140 29984 tablet_service.cc:1467] Processing CreateTablet for tablet a08467832a3043e8afd1d46dc5b4053d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=92b55a490df34e4695ec552b8fa8dee4]), partition=
I20240503 02:13:12.513547 29984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a08467832a3043e8afd1d46dc5b4053d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:12.514448 2860 tablet_bootstrap.cc:492] T a08467832a3043e8afd1d46dc5b4053d P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:12.516709 2860 tablet_bootstrap.cc:654] T a08467832a3043e8afd1d46dc5b4053d P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:12.517493 2767 tablet_bootstrap.cc:492] T a08467832a3043e8afd1d46dc5b4053d P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:13:12.519294 2767 tablet_bootstrap.cc:654] T a08467832a3043e8afd1d46dc5b4053d P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:12.520835 2862 tablet_bootstrap.cc:492] T a08467832a3043e8afd1d46dc5b4053d P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:13:12.522529 2862 tablet_bootstrap.cc:654] T a08467832a3043e8afd1d46dc5b4053d P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:12.523263 2860 tablet_bootstrap.cc:492] T a08467832a3043e8afd1d46dc5b4053d P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:12.523476 2860 ts_tablet_manager.cc:1367] T a08467832a3043e8afd1d46dc5b4053d P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240503 02:13:12.524230 2862 tablet_bootstrap.cc:492] T a08467832a3043e8afd1d46dc5b4053d P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:13:12.524266 2860 raft_consensus.cc:348] T a08467832a3043e8afd1d46dc5b4053d P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:12.524461 2862 ts_tablet_manager.cc:1367] T a08467832a3043e8afd1d46dc5b4053d P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:13:12.524520 2860 raft_consensus.cc:374] T a08467832a3043e8afd1d46dc5b4053d P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:12.524626 2860 raft_consensus.cc:729] T a08467832a3043e8afd1d46dc5b4053d P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:12.524914 2860 consensus_queue.cc:260] T a08467832a3043e8afd1d46dc5b4053d P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:12.525298 2862 raft_consensus.cc:348] T a08467832a3043e8afd1d46dc5b4053d P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:12.525553 2862 raft_consensus.cc:374] T a08467832a3043e8afd1d46dc5b4053d P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:12.525660 2862 raft_consensus.cc:729] T a08467832a3043e8afd1d46dc5b4053d P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:13:12.525701 2860 ts_tablet_manager.cc:1398] T a08467832a3043e8afd1d46dc5b4053d P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:12.525939 2862 consensus_queue.cc:260] T a08467832a3043e8afd1d46dc5b4053d P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:12.526515 2862 ts_tablet_manager.cc:1398] T a08467832a3043e8afd1d46dc5b4053d P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:13:12.532828 2767 tablet_bootstrap.cc:492] T a08467832a3043e8afd1d46dc5b4053d P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:13:12.533015 2767 ts_tablet_manager.cc:1367] T a08467832a3043e8afd1d46dc5b4053d P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.002s
I20240503 02:13:12.534303 2767 raft_consensus.cc:348] T a08467832a3043e8afd1d46dc5b4053d P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:12.534859 2767 raft_consensus.cc:374] T a08467832a3043e8afd1d46dc5b4053d P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:12.534989 2767 raft_consensus.cc:729] T a08467832a3043e8afd1d46dc5b4053d P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:13:12.535068 2745 consensus_queue.cc:1035] T ae4373a66af14c8f8a633963dffd2261 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:12.535333 2767 consensus_queue.cc:260] T a08467832a3043e8afd1d46dc5b4053d P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:12.536612 2767 ts_tablet_manager.cc:1398] T a08467832a3043e8afd1d46dc5b4053d P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:13:12.541368 2745 consensus_queue.cc:1035] T ae4373a66af14c8f8a633963dffd2261 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:12.570997 2794 consensus_queue.cc:1035] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:12.577840 2794 consensus_queue.cc:1035] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:12.579367 2020 raft_consensus.cc:482] T a08467832a3043e8afd1d46dc5b4053d P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:12.579550 2020 raft_consensus.cc:504] T a08467832a3043e8afd1d46dc5b4053d P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:12.580322 2020 leader_election.cc:288] T a08467832a3043e8afd1d46dc5b4053d P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:12.581140 30136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a08467832a3043e8afd1d46dc5b4053d" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:13:12.581408 30136 raft_consensus.cc:2454] T a08467832a3043e8afd1d46dc5b4053d P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:13:12.581869 29939 leader_election.cc:302] T a08467832a3043e8afd1d46dc5b4053d P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:13:12.582348 2810 raft_consensus.cc:2781] T a08467832a3043e8afd1d46dc5b4053d P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:12.582437 2810 raft_consensus.cc:482] T a08467832a3043e8afd1d46dc5b4053d P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:12.582494 2810 raft_consensus.cc:3037] T a08467832a3043e8afd1d46dc5b4053d P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:12.582413 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a08467832a3043e8afd1d46dc5b4053d" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:13:12.582648 307 raft_consensus.cc:2454] T a08467832a3043e8afd1d46dc5b4053d P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:13:12.585151 2810 raft_consensus.cc:504] T a08467832a3043e8afd1d46dc5b4053d P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:12.586213 30136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a08467832a3043e8afd1d46dc5b4053d" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:13:12.586477 30136 raft_consensus.cc:3037] T a08467832a3043e8afd1d46dc5b4053d P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:12.586634 2810 leader_election.cc:288] T a08467832a3043e8afd1d46dc5b4053d P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Requested vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:12.588798 30136 raft_consensus.cc:2454] T a08467832a3043e8afd1d46dc5b4053d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:13:12.589282 29939 leader_election.cc:302] T a08467832a3043e8afd1d46dc5b4053d P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:13:12.589327 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a08467832a3043e8afd1d46dc5b4053d" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:13:12.589515 307 raft_consensus.cc:3037] T a08467832a3043e8afd1d46dc5b4053d P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:12.589694 2810 raft_consensus.cc:2781] T a08467832a3043e8afd1d46dc5b4053d P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:12.589870 2810 raft_consensus.cc:686] T a08467832a3043e8afd1d46dc5b4053d P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Becoming Leader. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Running, Role: LEADER
I20240503 02:13:12.590240 2810 consensus_queue.cc:237] T a08467832a3043e8afd1d46dc5b4053d P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:12.592509 307 raft_consensus.cc:2454] T a08467832a3043e8afd1d46dc5b4053d P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:13:12.606009 29730 catalog_manager.cc:5484] T a08467832a3043e8afd1d46dc5b4053d P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: term changed from 0 to 1, leader changed from <none> to 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2). New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:12.715971 29984 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:12.716915 29732 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:13:12.721261 29853 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:12.721222 29725 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_extra_table_32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:12.736433 547 tablet_service.cc:1467] Processing CreateTablet for tablet 4055b2f8d72041acaa81e6e98db9042c (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=567dfe3064ef40a99eede42015bd1a35]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:12.737489 547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4055b2f8d72041acaa81e6e98db9042c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:12.743079 29853 tablet_service.cc:1467] Processing CreateTablet for tablet 4055b2f8d72041acaa81e6e98db9042c (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=567dfe3064ef40a99eede42015bd1a35]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:12.743534 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4055b2f8d72041acaa81e6e98db9042c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:12.751533 32755 tablet_service.cc:1467] Processing CreateTablet for tablet 4055b2f8d72041acaa81e6e98db9042c (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=567dfe3064ef40a99eede42015bd1a35]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:12.752038 32755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4055b2f8d72041acaa81e6e98db9042c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:12.757669 29984 tablet_service.cc:1467] Processing CreateTablet for tablet 368d683f40cd4d0994e7bbe90e045d16 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=567dfe3064ef40a99eede42015bd1a35]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:12.758149 29984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 368d683f40cd4d0994e7bbe90e045d16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:12.763996 30245 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:13:12.766183 32754 tablet_service.cc:1467] Processing CreateTablet for tablet 368d683f40cd4d0994e7bbe90e045d16 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=567dfe3064ef40a99eede42015bd1a35]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:12.766515 30245 tablet_service.cc:1467] Processing CreateTablet for tablet 368d683f40cd4d0994e7bbe90e045d16 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=567dfe3064ef40a99eede42015bd1a35]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:12.766635 32754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 368d683f40cd4d0994e7bbe90e045d16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:12.766893 30245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 368d683f40cd4d0994e7bbe90e045d16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:12.769850 2917 tablet_bootstrap.cc:492] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:13:12.774989 2917 tablet_bootstrap.cc:654] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:12.777207 2860 tablet_bootstrap.cc:492] T 368d683f40cd4d0994e7bbe90e045d16 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:12.779461 2860 tablet_bootstrap.cc:654] T 368d683f40cd4d0994e7bbe90e045d16 P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:12.787817 30116 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:12.787745 416 tablet_service.cc:1467] Processing CreateTablet for tablet 58e1aaecf47944569296fc1e0780efb5 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=567dfe3064ef40a99eede42015bd1a35]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:12.788218 416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58e1aaecf47944569296fc1e0780efb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:12.792488 2862 tablet_bootstrap.cc:492] T 368d683f40cd4d0994e7bbe90e045d16 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:13:12.794150 2862 tablet_bootstrap.cc:654] T 368d683f40cd4d0994e7bbe90e045d16 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:12.798903 2860 tablet_bootstrap.cc:492] T 368d683f40cd4d0994e7bbe90e045d16 P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:12.799252 2860 ts_tablet_manager.cc:1367] T 368d683f40cd4d0994e7bbe90e045d16 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.004s
I20240503 02:13:12.800264 2860 raft_consensus.cc:348] T 368d683f40cd4d0994e7bbe90e045d16 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:12.800611 2860 raft_consensus.cc:374] T 368d683f40cd4d0994e7bbe90e045d16 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:12.800756 2860 raft_consensus.cc:729] T 368d683f40cd4d0994e7bbe90e045d16 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:12.801328 2860 consensus_queue.cc:260] T 368d683f40cd4d0994e7bbe90e045d16 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:12.801602 2604 raft_consensus.cc:482] T 368d683f40cd4d0994e7bbe90e045d16 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:12.801777 2604 raft_consensus.cc:504] T 368d683f40cd4d0994e7bbe90e045d16 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:12.802562 2604 leader_election.cc:288] T 368d683f40cd4d0994e7bbe90e045d16 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:13:12.804215 2860 ts_tablet_manager.cc:1398] T 368d683f40cd4d0994e7bbe90e045d16 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 02:13:12.826495 32620 tablet_service.cc:1467] Processing CreateTablet for tablet 58e1aaecf47944569296fc1e0780efb5 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=567dfe3064ef40a99eede42015bd1a35]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:12.827008 32620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58e1aaecf47944569296fc1e0780efb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:12.828410 2862 tablet_bootstrap.cc:492] T 368d683f40cd4d0994e7bbe90e045d16 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:13:12.828600 2862 ts_tablet_manager.cc:1367] T 368d683f40cd4d0994e7bbe90e045d16 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.007s
I20240503 02:13:12.829483 2862 raft_consensus.cc:348] T 368d683f40cd4d0994e7bbe90e045d16 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:12.829813 2862 raft_consensus.cc:374] T 368d683f40cd4d0994e7bbe90e045d16 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:12.830188 2862 raft_consensus.cc:729] T 368d683f40cd4d0994e7bbe90e045d16 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:13:12.830389 29853 tablet_service.cc:1467] Processing CreateTablet for tablet 58e1aaecf47944569296fc1e0780efb5 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=567dfe3064ef40a99eede42015bd1a35]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:12.830500 2862 consensus_queue.cc:260] T 368d683f40cd4d0994e7bbe90e045d16 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:12.830845 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58e1aaecf47944569296fc1e0780efb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:12.831434 2862 ts_tablet_manager.cc:1398] T 368d683f40cd4d0994e7bbe90e045d16 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:12.888296 2925 tablet_bootstrap.cc:492] T 368d683f40cd4d0994e7bbe90e045d16 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:13:12.888327 417 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:13:12.896018 2860 tablet_bootstrap.cc:492] T 4055b2f8d72041acaa81e6e98db9042c P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:12.897799 2860 tablet_bootstrap.cc:654] T 4055b2f8d72041acaa81e6e98db9042c P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:12.903393 2860 tablet_bootstrap.cc:492] T 4055b2f8d72041acaa81e6e98db9042c P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:12.906445 2860 ts_tablet_manager.cc:1367] T 4055b2f8d72041acaa81e6e98db9042c P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20240503 02:13:12.907567 2860 raft_consensus.cc:348] T 4055b2f8d72041acaa81e6e98db9042c P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:12.907858 2860 raft_consensus.cc:374] T 4055b2f8d72041acaa81e6e98db9042c P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:12.907943 2860 raft_consensus.cc:729] T 4055b2f8d72041acaa81e6e98db9042c P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:12.908334 2860 consensus_queue.cc:260] T 4055b2f8d72041acaa81e6e98db9042c P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:12.908982 2860 ts_tablet_manager.cc:1398] T 4055b2f8d72041acaa81e6e98db9042c P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:13:12.919126 2926 tablet_bootstrap.cc:492] T 4055b2f8d72041acaa81e6e98db9042c P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:12.920244 2927 tablet_bootstrap.cc:492] T 58e1aaecf47944569296fc1e0780efb5 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:12.921816 2929 tablet_bootstrap.cc:492] T 58e1aaecf47944569296fc1e0780efb5 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:12.935778 2925 tablet_bootstrap.cc:654] T 368d683f40cd4d0994e7bbe90e045d16 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:12.946266 30135 raft_consensus.cc:1261] T a08467832a3043e8afd1d46dc5b4053d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:12.946820 2020 consensus_queue.cc:1035] T a08467832a3043e8afd1d46dc5b4053d P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:12.948062 2927 tablet_bootstrap.cc:654] T 58e1aaecf47944569296fc1e0780efb5 P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:12.952328 2926 tablet_bootstrap.cc:654] T 4055b2f8d72041acaa81e6e98db9042c P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:12.956781 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "368d683f40cd4d0994e7bbe90e045d16" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:13:12.957006 30005 raft_consensus.cc:2454] T 368d683f40cd4d0994e7bbe90e045d16 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:13:12.958226 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "368d683f40cd4d0994e7bbe90e045d16" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:13:12.962190 307 raft_consensus.cc:1261] T a08467832a3043e8afd1d46dc5b4053d P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:12.967272 2020 consensus_queue.cc:1035] T a08467832a3043e8afd1d46dc5b4053d P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:12.974030 32620 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:13.004850 29725 catalog_manager.cc:5576] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679) does not have the latest schema for tablet a08467832a3043e8afd1d46dc5b4053d (table rebalancer_dynamic_table [id=92b55a490df34e4695ec552b8fa8dee4]). Expected version 1 got 0
I20240503 02:13:13.023130 2936 tablet.cc:1722] T a08467832a3043e8afd1d46dc5b4053d P a9094240860e43bfb129193f3bfd5be6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:13.026782 2936 tablet.cc:1620] T a08467832a3043e8afd1d46dc5b4053d P a9094240860e43bfb129193f3bfd5be6: MemRowSet was empty: no flush needed.
I20240503 02:13:13.027096 29725 catalog_manager.cc:5576] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679) does not have the latest schema for tablet a08467832a3043e8afd1d46dc5b4053d (table rebalancer_dynamic_table [id=92b55a490df34e4695ec552b8fa8dee4]). Expected version 1 got 0
I20240503 02:13:13.042623 2935 tablet.cc:1722] T a08467832a3043e8afd1d46dc5b4053d P 155735c7f8b14997ac4c770aee2a9401: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:13.043164 2935 tablet.cc:1620] T a08467832a3043e8afd1d46dc5b4053d P 155735c7f8b14997ac4c770aee2a9401: MemRowSet was empty: no flush needed.
I20240503 02:13:13.044811 2929 tablet_bootstrap.cc:654] T 58e1aaecf47944569296fc1e0780efb5 P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:13.046753 29725 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=92b55a490df34e4695ec552b8fa8dee4] alter complete (version 1)
I20240503 02:13:13.008196 547 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:13:13.051971 2926 tablet_bootstrap.cc:492] T 4055b2f8d72041acaa81e6e98db9042c P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:13.052867 2917 tablet_bootstrap.cc:492] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:13:13.058262 2926 ts_tablet_manager.cc:1367] T 4055b2f8d72041acaa81e6e98db9042c P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.139s user 0.004s sys 0.000s
I20240503 02:13:13.061611 2926 raft_consensus.cc:348] T 4055b2f8d72041acaa81e6e98db9042c P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
W20240503 02:13:13.053567 32708 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.8:42233, user_credentials={real_user=slave}} blocked reactor thread for 87535.1us
I20240503 02:13:13.063288 29725 catalog_manager.cc:5576] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975) does not have the latest schema for tablet a08467832a3043e8afd1d46dc5b4053d (table rebalancer_dynamic_table [id=92b55a490df34e4695ec552b8fa8dee4]). Expected version 1 got 0
I20240503 02:13:13.064110 2926 raft_consensus.cc:374] T 4055b2f8d72041acaa81e6e98db9042c P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:13.064524 2926 raft_consensus.cc:729] T 4055b2f8d72041acaa81e6e98db9042c P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:13.065128 2917 ts_tablet_manager.cc:1367] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.295s user 0.008s sys 0.023s
I20240503 02:13:13.065163 2926 consensus_queue.cc:260] T 4055b2f8d72041acaa81e6e98db9042c P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:13.073956 2917 raft_consensus.cc:348] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:13.074770 2926 ts_tablet_manager.cc:1398] T 4055b2f8d72041acaa81e6e98db9042c P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240503 02:13:13.079435 2929 tablet_bootstrap.cc:492] T 58e1aaecf47944569296fc1e0780efb5 P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:13.079672 2929 ts_tablet_manager.cc:1367] T 58e1aaecf47944569296fc1e0780efb5 P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.158s user 0.004s sys 0.000s
I20240503 02:13:13.080459 2929 raft_consensus.cc:348] T 58e1aaecf47944569296fc1e0780efb5 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:13.080745 2929 raft_consensus.cc:374] T 58e1aaecf47944569296fc1e0780efb5 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:13.080863 2929 raft_consensus.cc:729] T 58e1aaecf47944569296fc1e0780efb5 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:13.081161 2929 consensus_queue.cc:260] T 58e1aaecf47944569296fc1e0780efb5 P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:13.081755 2929 ts_tablet_manager.cc:1398] T 58e1aaecf47944569296fc1e0780efb5 P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240503 02:13:13.077416 32709 leader_election.cc:341] T 368d683f40cd4d0994e7bbe90e045d16 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:13.090272 32709 leader_election.cc:302] T 368d683f40cd4d0994e7bbe90e045d16 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 8a666e798fe8477d8dd1e51b385d5ead; no voters: 1f10f80f13c84e80b03eda7085a3b012
I20240503 02:13:13.096388 2917 raft_consensus.cc:374] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:13.096736 2787 raft_consensus.cc:2781] T 368d683f40cd4d0994e7bbe90e045d16 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:13.096868 2787 raft_consensus.cc:482] T 368d683f40cd4d0994e7bbe90e045d16 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:13.096984 2787 raft_consensus.cc:3037] T 368d683f40cd4d0994e7bbe90e045d16 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:13.099906 2787 raft_consensus.cc:504] T 368d683f40cd4d0994e7bbe90e045d16 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:13.100523 2787 leader_election.cc:288] T 368d683f40cd4d0994e7bbe90e045d16 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Requested vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
W20240503 02:13:13.109200 29940 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.6:39975, user_credentials={real_user=slave}} blocked reactor thread for 44688.1us
I20240503 02:13:13.112740 2917 raft_consensus.cc:729] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:13:13.114377 2917 consensus_queue.cc:260] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:13.118367 2925 tablet_bootstrap.cc:492] T 368d683f40cd4d0994e7bbe90e045d16 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:13:13.118554 2925 ts_tablet_manager.cc:1367] T 368d683f40cd4d0994e7bbe90e045d16 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.230s user 0.007s sys 0.010s
I20240503 02:13:13.119243 2925 raft_consensus.cc:348] T 368d683f40cd4d0994e7bbe90e045d16 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:13.119550 2925 raft_consensus.cc:374] T 368d683f40cd4d0994e7bbe90e045d16 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:13.119660 2925 raft_consensus.cc:729] T 368d683f40cd4d0994e7bbe90e045d16 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:13:13.119937 2925 consensus_queue.cc:260] T 368d683f40cd4d0994e7bbe90e045d16 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:13.120502 2925 ts_tablet_manager.cc:1398] T 368d683f40cd4d0994e7bbe90e045d16 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:13:13.141664 2945 tablet.cc:1722] T a08467832a3043e8afd1d46dc5b4053d P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:13.142413 2945 tablet.cc:1620] T a08467832a3043e8afd1d46dc5b4053d P 8a666e798fe8477d8dd1e51b385d5ead: MemRowSet was empty: no flush needed.
I20240503 02:13:13.149428 2927 tablet_bootstrap.cc:492] T 58e1aaecf47944569296fc1e0780efb5 P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:13.149576 2927 ts_tablet_manager.cc:1367] T 58e1aaecf47944569296fc1e0780efb5 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.229s user 0.020s sys 0.007s
I20240503 02:13:13.150318 2927 raft_consensus.cc:348] T 58e1aaecf47944569296fc1e0780efb5 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:13.150532 2927 raft_consensus.cc:374] T 58e1aaecf47944569296fc1e0780efb5 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:13.150511 29725 catalog_manager.cc:5576] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975) does not have the latest schema for tablet a08467832a3043e8afd1d46dc5b4053d (table rebalancer_dynamic_table [id=92b55a490df34e4695ec552b8fa8dee4]). Expected version 1 got 0
I20240503 02:13:13.150629 2927 raft_consensus.cc:729] T 58e1aaecf47944569296fc1e0780efb5 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:13.150883 2927 consensus_queue.cc:260] T 58e1aaecf47944569296fc1e0780efb5 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:13.152209 1484 raft_consensus.cc:482] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:13.152503 2927 ts_tablet_manager.cc:1398] T 58e1aaecf47944569296fc1e0780efb5 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:13:13.152431 1484 raft_consensus.cc:504] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:13.153810 1484 leader_election.cc:288] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:13.154299 566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4055b2f8d72041acaa81e6e98db9042c" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:13.154551 566 raft_consensus.cc:2454] T 4055b2f8d72041acaa81e6e98db9042c P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:13:13.155066 29808 leader_election.cc:302] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:13.155431 1484 raft_consensus.cc:2781] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:13.155539 1484 raft_consensus.cc:482] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:13.155664 1484 raft_consensus.cc:3037] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:13.145153 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "368d683f40cd4d0994e7bbe90e045d16" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:13:13.156270 30265 raft_consensus.cc:3037] T 368d683f40cd4d0994e7bbe90e045d16 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:13.158674 1484 raft_consensus.cc:504] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:13.159899 566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4055b2f8d72041acaa81e6e98db9042c" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:13:13.160120 566 raft_consensus.cc:3037] T 4055b2f8d72041acaa81e6e98db9042c P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:13.161106 30265 raft_consensus.cc:2454] T 368d683f40cd4d0994e7bbe90e045d16 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:13:13.163054 566 raft_consensus.cc:2454] T 4055b2f8d72041acaa81e6e98db9042c P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:13:13.164673 2917 ts_tablet_manager.cc:1398] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.095s user 0.002s sys 0.000s
I20240503 02:13:13.166206 29808 leader_election.cc:302] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:13.166636 2755 raft_consensus.cc:2781] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:13.169365 29732 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
W20240503 02:13:13.171927 29939 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.3:42679, user_credentials={real_user=slave}} blocked reactor thread for 132164us
I20240503 02:13:13.172868 2917 tablet_bootstrap.cc:492] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:13:13.177649 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4055b2f8d72041acaa81e6e98db9042c" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:13:13.178040 305 raft_consensus.cc:2454] T 4055b2f8d72041acaa81e6e98db9042c P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:13:13.179399 32709 leader_election.cc:302] T 368d683f40cd4d0994e7bbe90e045d16 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, 8a666e798fe8477d8dd1e51b385d5ead; no voters:
I20240503 02:13:13.179783 2931 raft_consensus.cc:2781] T 368d683f40cd4d0994e7bbe90e045d16 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:13.179929 2931 raft_consensus.cc:686] T 368d683f40cd4d0994e7bbe90e045d16 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Becoming Leader. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Running, Role: LEADER
I20240503 02:13:13.180212 2931 consensus_queue.cc:237] T 368d683f40cd4d0994e7bbe90e045d16 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:13.180711 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "368d683f40cd4d0994e7bbe90e045d16" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:13:13.180900 30004 raft_consensus.cc:3037] T 368d683f40cd4d0994e7bbe90e045d16 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:13.181123 2917 tablet_bootstrap.cc:654] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:13.183733 30004 raft_consensus.cc:2454] T 368d683f40cd4d0994e7bbe90e045d16 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:13:13.188133 2935 tablet.cc:1722] T a08467832a3043e8afd1d46dc5b4053d P 155735c7f8b14997ac4c770aee2a9401: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:13.188585 2935 tablet.cc:1620] T a08467832a3043e8afd1d46dc5b4053d P 155735c7f8b14997ac4c770aee2a9401: MemRowSet was empty: no flush needed.
I20240503 02:13:13.191560 1484 leader_election.cc:288] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Requested vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:13.192730 2936 tablet.cc:1722] T a08467832a3043e8afd1d46dc5b4053d P a9094240860e43bfb129193f3bfd5be6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:13.193264 2936 tablet.cc:1620] T a08467832a3043e8afd1d46dc5b4053d P a9094240860e43bfb129193f3bfd5be6: MemRowSet was empty: no flush needed.
I20240503 02:13:13.194031 2945 tablet.cc:1722] T a08467832a3043e8afd1d46dc5b4053d P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:13.194448 2945 tablet.cc:1620] T a08467832a3043e8afd1d46dc5b4053d P 8a666e798fe8477d8dd1e51b385d5ead: MemRowSet was empty: no flush needed.
I20240503 02:13:13.201965 2917 tablet_bootstrap.cc:492] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:13:13.202191 2917 ts_tablet_manager.cc:1367] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.004s
I20240503 02:13:13.203240 2917 raft_consensus.cc:348] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:13.203583 2917 raft_consensus.cc:374] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:13.203717 2917 raft_consensus.cc:729] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:13:13.204025 2917 consensus_queue.cc:260] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:13.204622 2917 ts_tablet_manager.cc:1398] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:13:13.217927 29725 catalog_manager.cc:5576] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975) does not have the latest schema for tablet a08467832a3043e8afd1d46dc5b4053d (table rebalancer_dynamic_table [id=92b55a490df34e4695ec552b8fa8dee4]). Expected version 2 got 1
W20240503 02:13:13.218684 29809 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.2:37873, user_credentials={real_user=slave}} blocked reactor thread for 45370us
I20240503 02:13:13.219453 29732 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=92b55a490df34e4695ec552b8fa8dee4] alter complete (version 2)
I20240503 02:13:13.219645 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4055b2f8d72041acaa81e6e98db9042c" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:13:13.219906 307 raft_consensus.cc:3037] T 4055b2f8d72041acaa81e6e98db9042c P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:13.223906 307 raft_consensus.cc:2454] T 4055b2f8d72041acaa81e6e98db9042c P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
W20240503 02:13:13.225883 32709 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.108.4:33935, user_credentials={real_user=slave}} blocked reactor thread for 46493.4us
I20240503 02:13:13.227003 2755 raft_consensus.cc:686] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Becoming Leader. State: Replica: 16440e365430405fb10366ab70e2b429, State: Running, Role: LEADER
I20240503 02:13:13.227331 2755 consensus_queue.cc:237] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:13.234417 29725 catalog_manager.cc:5484] T 368d683f40cd4d0994e7bbe90e045d16 P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6). New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:13.235672 32755 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240503 02:13:13.235824 502 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.5:35595, user_credentials={real_user=slave}} blocked reactor thread for 92139.7us
I20240503 02:13:13.244294 29725 catalog_manager.cc:5484] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16440e365430405fb10366ab70e2b429 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:13.245397 2755 raft_consensus.cc:482] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:13.245611 2755 raft_consensus.cc:504] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:13.246213 2755 leader_election.cc:288] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:13:13.247040 435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58e1aaecf47944569296fc1e0780efb5" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:13:13.247305 435 raft_consensus.cc:2454] T 58e1aaecf47944569296fc1e0780efb5 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:13:13.248100 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58e1aaecf47944569296fc1e0780efb5" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:13:13.248345 32640 raft_consensus.cc:2454] T 58e1aaecf47944569296fc1e0780efb5 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:13:13.249321 29809 leader_election.cc:302] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, 9a4d92bc25374fb3960fde7d10be309d; no voters:
I20240503 02:13:13.249570 1484 raft_consensus.cc:2781] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:13.249641 1484 raft_consensus.cc:482] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:13.249703 1484 raft_consensus.cc:3037] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:13.252398 1484 raft_consensus.cc:504] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:13.253207 1484 leader_election.cc:288] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Requested vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:13:13.255653 435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58e1aaecf47944569296fc1e0780efb5" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:13:13.255920 435 raft_consensus.cc:3037] T 58e1aaecf47944569296fc1e0780efb5 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:13.259045 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58e1aaecf47944569296fc1e0780efb5" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d"
I20240503 02:13:13.259274 32640 raft_consensus.cc:3037] T 58e1aaecf47944569296fc1e0780efb5 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:13.260494 435 raft_consensus.cc:2454] T 58e1aaecf47944569296fc1e0780efb5 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:13:13.262197 32640 raft_consensus.cc:2454] T 58e1aaecf47944569296fc1e0780efb5 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:13:13.262534 29809 leader_election.cc:302] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:13.262811 1484 raft_consensus.cc:2781] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:13.262954 1484 raft_consensus.cc:686] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Becoming Leader. State: Replica: 16440e365430405fb10366ab70e2b429, State: Running, Role: LEADER
I20240503 02:13:13.263192 1484 consensus_queue.cc:237] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:13.266712 29725 catalog_manager.cc:5484] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16440e365430405fb10366ab70e2b429 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:13.322165 29730 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:13:13.332445 2935 tablet.cc:1722] T a08467832a3043e8afd1d46dc5b4053d P 155735c7f8b14997ac4c770aee2a9401: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:13.333154 2936 tablet.cc:1722] T a08467832a3043e8afd1d46dc5b4053d P a9094240860e43bfb129193f3bfd5be6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:13.333873 30004 consensus_queue.cc:237] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:13.336041 435 raft_consensus.cc:1261] T 35a5a7f4fe184b9fadc94c98b61deaf0 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:13.337215 2942 consensus_queue.cc:1035] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:13.337299 29872 raft_consensus.cc:1261] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:13.337396 2945 tablet.cc:1722] T a08467832a3043e8afd1d46dc5b4053d P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:13.338982 2937 consensus_queue.cc:1035] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:13.344494 29725 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table_new_name [id=92b55a490df34e4695ec552b8fa8dee4] alter complete (version 3)
I20240503 02:13:13.344451 2942 raft_consensus.cc:2932] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:13.346881 29872 raft_consensus.cc:2932] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:13.349336 437 raft_consensus.cc:2932] T 35a5a7f4fe184b9fadc94c98b61deaf0 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:13.352533 29730 catalog_manager.cc:5484] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 16440e365430405fb10366ab70e2b429 reported cstate change: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:13.358036 30116 tablet_service.cc:1514] Processing DeleteTablet for tablet 35a5a7f4fe184b9fadc94c98b61deaf0 with delete_type TABLET_DATA_TOMBSTONED (TS a9094240860e43bfb129193f3bfd5be6 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44812
I20240503 02:13:13.375702 2975 tablet_replica.cc:330] T 35a5a7f4fe184b9fadc94c98b61deaf0 P a9094240860e43bfb129193f3bfd5be6: stopping tablet replica
I20240503 02:13:13.376103 2975 raft_consensus.cc:2229] T 35a5a7f4fe184b9fadc94c98b61deaf0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:13.376461 2975 raft_consensus.cc:2258] T 35a5a7f4fe184b9fadc94c98b61deaf0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:13.377312 2975 ts_tablet_manager.cc:1875] T 35a5a7f4fe184b9fadc94c98b61deaf0 P a9094240860e43bfb129193f3bfd5be6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:13.386595 2975 ts_tablet_manager.cc:1888] T 35a5a7f4fe184b9fadc94c98b61deaf0 P a9094240860e43bfb129193f3bfd5be6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:13:13.386811 2975 log.cc:1196] T 35a5a7f4fe184b9fadc94c98b61deaf0 P a9094240860e43bfb129193f3bfd5be6: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/wals/35a5a7f4fe184b9fadc94c98b61deaf0
I20240503 02:13:13.387665 29718 catalog_manager.cc:4830] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): tablet 35a5a7f4fe184b9fadc94c98b61deaf0 (table rebalance_test_table_3 [id=3131069dba4a4f9aa26d4c134fcb4704]) successfully deleted
I20240503 02:13:13.459391 29725 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:13.459604 29725 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:13.462934 29725 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet a08467832a3043e8afd1d46dc5b4053d
I20240503 02:13:13.463790 30116 tablet_service.cc:1514] Processing DeleteTablet for tablet a08467832a3043e8afd1d46dc5b4053d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:13 UTC) from {username='slave'} at 127.0.0.1:44812
I20240503 02:13:13.464164 2975 tablet_replica.cc:330] T a08467832a3043e8afd1d46dc5b4053d P a9094240860e43bfb129193f3bfd5be6: stopping tablet replica
I20240503 02:13:13.464376 2975 raft_consensus.cc:2229] T a08467832a3043e8afd1d46dc5b4053d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:13.464607 2975 raft_consensus.cc:2258] T a08467832a3043e8afd1d46dc5b4053d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:13.465780 2975 ts_tablet_manager.cc:1875] T a08467832a3043e8afd1d46dc5b4053d P a9094240860e43bfb129193f3bfd5be6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:13.469666 32755 tablet_service.cc:1514] Processing DeleteTablet for tablet a08467832a3043e8afd1d46dc5b4053d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:13 UTC) from {username='slave'} at 127.0.0.1:46360
I20240503 02:13:13.470291 29984 tablet_service.cc:1514] Processing DeleteTablet for tablet a08467832a3043e8afd1d46dc5b4053d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:13 UTC) from {username='slave'} at 127.0.0.1:52158
I20240503 02:13:13.470873 2989 tablet_replica.cc:330] T a08467832a3043e8afd1d46dc5b4053d P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:13:13.471132 2989 raft_consensus.cc:2229] T a08467832a3043e8afd1d46dc5b4053d P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:13.471506 2989 raft_consensus.cc:2258] T a08467832a3043e8afd1d46dc5b4053d P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:13.473723 2988 tablet_replica.cc:330] T a08467832a3043e8afd1d46dc5b4053d P 8a666e798fe8477d8dd1e51b385d5ead: stopping tablet replica
I20240503 02:13:13.473824 2989 ts_tablet_manager.cc:1875] T a08467832a3043e8afd1d46dc5b4053d P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:13.473939 2988 raft_consensus.cc:2229] T a08467832a3043e8afd1d46dc5b4053d P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:13.474195 2988 raft_consensus.cc:2258] T a08467832a3043e8afd1d46dc5b4053d P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:13.475145 2988 ts_tablet_manager.cc:1875] T a08467832a3043e8afd1d46dc5b4053d P 8a666e798fe8477d8dd1e51b385d5ead: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:13.476593 2975 ts_tablet_manager.cc:1888] T a08467832a3043e8afd1d46dc5b4053d P a9094240860e43bfb129193f3bfd5be6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:13.476877 2975 log.cc:1196] T a08467832a3043e8afd1d46dc5b4053d P a9094240860e43bfb129193f3bfd5be6: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/wals/a08467832a3043e8afd1d46dc5b4053d
I20240503 02:13:13.477483 2975 ts_tablet_manager.cc:1909] T a08467832a3043e8afd1d46dc5b4053d P a9094240860e43bfb129193f3bfd5be6: Deleting consensus metadata
I20240503 02:13:13.478754 29718 catalog_manager.cc:4830] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): tablet a08467832a3043e8afd1d46dc5b4053d (table rebalancer_dynamic_table_new_name [id=92b55a490df34e4695ec552b8fa8dee4]) successfully deleted
I20240503 02:13:13.488373 2988 ts_tablet_manager.cc:1888] T a08467832a3043e8afd1d46dc5b4053d P 8a666e798fe8477d8dd1e51b385d5ead: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:13.488605 2988 log.cc:1196] T a08467832a3043e8afd1d46dc5b4053d P 8a666e798fe8477d8dd1e51b385d5ead: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal/wals/a08467832a3043e8afd1d46dc5b4053d
I20240503 02:13:13.489145 2988 ts_tablet_manager.cc:1909] T a08467832a3043e8afd1d46dc5b4053d P 8a666e798fe8477d8dd1e51b385d5ead: Deleting consensus metadata
I20240503 02:13:13.490339 29719 catalog_manager.cc:4830] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): tablet a08467832a3043e8afd1d46dc5b4053d (table rebalancer_dynamic_table_new_name [id=92b55a490df34e4695ec552b8fa8dee4]) successfully deleted
I20240503 02:13:13.490825 2989 ts_tablet_manager.cc:1888] T a08467832a3043e8afd1d46dc5b4053d P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:13.490988 2989 log.cc:1196] T a08467832a3043e8afd1d46dc5b4053d P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/a08467832a3043e8afd1d46dc5b4053d
I20240503 02:13:13.491456 2989 ts_tablet_manager.cc:1909] T a08467832a3043e8afd1d46dc5b4053d P 155735c7f8b14997ac4c770aee2a9401: Deleting consensus metadata
I20240503 02:13:13.492637 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet a08467832a3043e8afd1d46dc5b4053d (table rebalancer_dynamic_table_new_name [id=92b55a490df34e4695ec552b8fa8dee4]) successfully deleted
I20240503 02:13:13.569478 29733 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:13:13.579133 416 tablet_service.cc:1467] Processing CreateTablet for tablet efd50e7bc6f04e43bba12e73bb6647db (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c388300a2a094bd4b9811c52202ec490]), partition=
I20240503 02:13:13.579411 32620 tablet_service.cc:1467] Processing CreateTablet for tablet efd50e7bc6f04e43bba12e73bb6647db (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c388300a2a094bd4b9811c52202ec490]), partition=
I20240503 02:13:13.579664 416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efd50e7bc6f04e43bba12e73bb6647db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:13.579795 32620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efd50e7bc6f04e43bba12e73bb6647db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:13.585144 32755 tablet_service.cc:1467] Processing CreateTablet for tablet efd50e7bc6f04e43bba12e73bb6647db (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c388300a2a094bd4b9811c52202ec490]), partition=
I20240503 02:13:13.585530 32755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efd50e7bc6f04e43bba12e73bb6647db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:13.589169 2927 tablet_bootstrap.cc:492] T efd50e7bc6f04e43bba12e73bb6647db P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:13.590814 2927 tablet_bootstrap.cc:654] T efd50e7bc6f04e43bba12e73bb6647db P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:13.599097 2931 consensus_queue.cc:1035] T 368d683f40cd4d0994e7bbe90e045d16 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:13.605976 2931 consensus_queue.cc:1035] T 368d683f40cd4d0994e7bbe90e045d16 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:13.609766 2755 consensus_queue.cc:1035] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:13.619982 1484 consensus_queue.cc:1035] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:13.649718 3027 tablet_bootstrap.cc:492] T efd50e7bc6f04e43bba12e73bb6647db P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:13.655164 3027 tablet_bootstrap.cc:654] T efd50e7bc6f04e43bba12e73bb6647db P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:13.666018 3026 tablet_bootstrap.cc:492] T efd50e7bc6f04e43bba12e73bb6647db P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:13.668274 2927 tablet_bootstrap.cc:492] T efd50e7bc6f04e43bba12e73bb6647db P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:13.668478 2927 ts_tablet_manager.cc:1367] T efd50e7bc6f04e43bba12e73bb6647db P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.079s user 0.015s sys 0.026s
I20240503 02:13:13.669413 2927 raft_consensus.cc:348] T efd50e7bc6f04e43bba12e73bb6647db P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:13.669667 2927 raft_consensus.cc:374] T efd50e7bc6f04e43bba12e73bb6647db P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:13.669754 2927 raft_consensus.cc:729] T efd50e7bc6f04e43bba12e73bb6647db P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:13.670861 2927 consensus_queue.cc:260] T efd50e7bc6f04e43bba12e73bb6647db P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:13.672343 3027 tablet_bootstrap.cc:492] T efd50e7bc6f04e43bba12e73bb6647db P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:13.672353 3026 tablet_bootstrap.cc:654] T efd50e7bc6f04e43bba12e73bb6647db P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:13.672502 3027 ts_tablet_manager.cc:1367] T efd50e7bc6f04e43bba12e73bb6647db P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.000s
I20240503 02:13:13.673539 3027 raft_consensus.cc:348] T efd50e7bc6f04e43bba12e73bb6647db P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:13.673758 3027 raft_consensus.cc:374] T efd50e7bc6f04e43bba12e73bb6647db P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:13.673861 3027 raft_consensus.cc:729] T efd50e7bc6f04e43bba12e73bb6647db P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:13.674160 3027 consensus_queue.cc:260] T efd50e7bc6f04e43bba12e73bb6647db P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:13.674764 3027 ts_tablet_manager.cc:1398] T efd50e7bc6f04e43bba12e73bb6647db P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:13:13.675407 2927 ts_tablet_manager.cc:1398] T efd50e7bc6f04e43bba12e73bb6647db P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240503 02:13:13.677964 1484 consensus_queue.cc:1035] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:13.683238 3026 tablet_bootstrap.cc:492] T efd50e7bc6f04e43bba12e73bb6647db P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:13.684083 3026 ts_tablet_manager.cc:1367] T efd50e7bc6f04e43bba12e73bb6647db P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.007s
I20240503 02:13:13.685299 3026 raft_consensus.cc:348] T efd50e7bc6f04e43bba12e73bb6647db P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:13.685593 3026 raft_consensus.cc:374] T efd50e7bc6f04e43bba12e73bb6647db P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:13.685716 3026 raft_consensus.cc:729] T efd50e7bc6f04e43bba12e73bb6647db P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:13.686221 3026 consensus_queue.cc:260] T efd50e7bc6f04e43bba12e73bb6647db P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:13.686578 2947 consensus_queue.cc:1035] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:13.686877 3026 ts_tablet_manager.cc:1398] T efd50e7bc6f04e43bba12e73bb6647db P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:13:13.740098 29853 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:13:13.752993 30245 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:13:13.762599 30116 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:13.768675 29984 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:13:13.798329 32755 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:13.833612 29725 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_extra_table_34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:13.838979 547 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:13:13.849313 29984 tablet_service.cc:1467] Processing CreateTablet for tablet 7a6f23bb933949e9868139608456130f (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=2274af2004634409884d3bcbca0009ca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:13.849808 29984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a6f23bb933949e9868139608456130f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:13.849752 29853 tablet_service.cc:1467] Processing CreateTablet for tablet 7a6f23bb933949e9868139608456130f (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=2274af2004634409884d3bcbca0009ca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:13.849843 32755 tablet_service.cc:1467] Processing CreateTablet for tablet eb0f4fe6b8684f329725ac2f7d5428f2 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=2274af2004634409884d3bcbca0009ca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:13.850160 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a6f23bb933949e9868139608456130f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:13.850327 32755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb0f4fe6b8684f329725ac2f7d5428f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:13.853366 29854 tablet_service.cc:1467] Processing CreateTablet for tablet eb0f4fe6b8684f329725ac2f7d5428f2 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=2274af2004634409884d3bcbca0009ca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:13.853726 29854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb0f4fe6b8684f329725ac2f7d5428f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:13.853835 32620 tablet_service.cc:1467] Processing CreateTablet for tablet 7a6f23bb933949e9868139608456130f (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=2274af2004634409884d3bcbca0009ca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:13.854164 547 tablet_service.cc:1467] Processing CreateTablet for tablet eb0f4fe6b8684f329725ac2f7d5428f2 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=2274af2004634409884d3bcbca0009ca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:13.854336 32620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a6f23bb933949e9868139608456130f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:13.854576 547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb0f4fe6b8684f329725ac2f7d5428f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:13.857998 32754 tablet_service.cc:1467] Processing CreateTablet for tablet e165c71f8b6645a6936b58b2a8da1842 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=2274af2004634409884d3bcbca0009ca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:13.858170 3027 tablet_bootstrap.cc:492] T eb0f4fe6b8684f329725ac2f7d5428f2 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:13.858412 32754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e165c71f8b6645a6936b58b2a8da1842. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:13.860003 3027 tablet_bootstrap.cc:654] T eb0f4fe6b8684f329725ac2f7d5428f2 P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:13.868052 548 tablet_service.cc:1467] Processing CreateTablet for tablet e165c71f8b6645a6936b58b2a8da1842 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=2274af2004634409884d3bcbca0009ca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:13.870976 3026 tablet_bootstrap.cc:492] T 7a6f23bb933949e9868139608456130f P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:13.873076 3026 tablet_bootstrap.cc:654] T 7a6f23bb933949e9868139608456130f P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:13.874122 3027 tablet_bootstrap.cc:492] T eb0f4fe6b8684f329725ac2f7d5428f2 P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:13.874298 3027 ts_tablet_manager.cc:1367] T eb0f4fe6b8684f329725ac2f7d5428f2 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.003s
I20240503 02:13:13.875108 3027 raft_consensus.cc:348] T eb0f4fe6b8684f329725ac2f7d5428f2 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:13.875347 3027 raft_consensus.cc:374] T eb0f4fe6b8684f329725ac2f7d5428f2 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:13.875427 3027 raft_consensus.cc:729] T eb0f4fe6b8684f329725ac2f7d5428f2 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:13.875664 3027 consensus_queue.cc:260] T eb0f4fe6b8684f329725ac2f7d5428f2 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:13.878418 2931 raft_consensus.cc:482] T eb0f4fe6b8684f329725ac2f7d5428f2 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:13.878585 2931 raft_consensus.cc:504] T eb0f4fe6b8684f329725ac2f7d5428f2 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:13.878996 30116 tablet_service.cc:1467] Processing CreateTablet for tablet e165c71f8b6645a6936b58b2a8da1842 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=2274af2004634409884d3bcbca0009ca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:13.879087 3027 ts_tablet_manager.cc:1398] T eb0f4fe6b8684f329725ac2f7d5428f2 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240503 02:13:13.879434 30116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e165c71f8b6645a6936b58b2a8da1842. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:13.879454 2931 leader_election.cc:288] T eb0f4fe6b8684f329725ac2f7d5428f2 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:13:13.880149 29872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb0f4fe6b8684f329725ac2f7d5428f2" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
W20240503 02:13:13.880648 32711 leader_election.cc:341] T eb0f4fe6b8684f329725ac2f7d5428f2 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:13.884120 548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e165c71f8b6645a6936b58b2a8da1842. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:13:13.896265 32708 leader_election.cc:341] T eb0f4fe6b8684f329725ac2f7d5428f2 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Not found: Tablet not found: eb0f4fe6b8684f329725ac2f7d5428f2
I20240503 02:13:13.896373 32620 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:13.896426 32708 leader_election.cc:302] T eb0f4fe6b8684f329725ac2f7d5428f2 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead; no voters: 16440e365430405fb10366ab70e2b429, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:13:13.896746 2931 raft_consensus.cc:2726] T eb0f4fe6b8684f329725ac2f7d5428f2 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:13.904600 416 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:13:13.918761 3055 tablet_bootstrap.cc:492] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:13:13.922191 3055 tablet_bootstrap.cc:654] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:13.924685 3027 tablet_bootstrap.cc:492] T e165c71f8b6645a6936b58b2a8da1842 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
W20240503 02:13:13.926296 371 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.5:35595, user_credentials={real_user=slave}} blocked reactor thread for 124964us
I20240503 02:13:13.926407 3027 tablet_bootstrap.cc:654] T e165c71f8b6645a6936b58b2a8da1842 P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:13.928993 3026 tablet_bootstrap.cc:492] T 7a6f23bb933949e9868139608456130f P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:13.929281 3026 ts_tablet_manager.cc:1367] T 7a6f23bb933949e9868139608456130f P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.058s user 0.009s sys 0.007s
I20240503 02:13:13.932873 3026 raft_consensus.cc:348] T 7a6f23bb933949e9868139608456130f P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:13.935263 3026 raft_consensus.cc:374] T 7a6f23bb933949e9868139608456130f P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:13.935385 3026 raft_consensus.cc:729] T 7a6f23bb933949e9868139608456130f P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:13.935727 3026 consensus_queue.cc:260] T 7a6f23bb933949e9868139608456130f P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:13.937119 3027 tablet_bootstrap.cc:492] T e165c71f8b6645a6936b58b2a8da1842 P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:13.937321 3027 ts_tablet_manager.cc:1367] T e165c71f8b6645a6936b58b2a8da1842 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20240503 02:13:13.938143 3027 raft_consensus.cc:348] T e165c71f8b6645a6936b58b2a8da1842 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:13.938410 3027 raft_consensus.cc:374] T e165c71f8b6645a6936b58b2a8da1842 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:13.938555 3027 raft_consensus.cc:729] T e165c71f8b6645a6936b58b2a8da1842 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:13.938817 3027 consensus_queue.cc:260] T e165c71f8b6645a6936b58b2a8da1842 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:13.939410 3027 ts_tablet_manager.cc:1398] T e165c71f8b6645a6936b58b2a8da1842 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:13:13.939999 3026 ts_tablet_manager.cc:1398] T 7a6f23bb933949e9868139608456130f P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:13:13.943538 3055 tablet_bootstrap.cc:492] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:13:13.944679 3055 ts_tablet_manager.cc:1367] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20240503 02:13:13.945586 3055 raft_consensus.cc:348] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:13.945886 3055 raft_consensus.cc:374] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:13.945992 3055 raft_consensus.cc:729] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:13:13.946291 3055 consensus_queue.cc:260] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:13.946854 3055 ts_tablet_manager.cc:1398] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:13:13.949903 2931 raft_consensus.cc:482] T efd50e7bc6f04e43bba12e73bb6647db P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:13.950073 2931 raft_consensus.cc:504] T efd50e7bc6f04e43bba12e73bb6647db P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:13.950801 2931 leader_election.cc:288] T efd50e7bc6f04e43bba12e73bb6647db P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:13.951375 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd50e7bc6f04e43bba12e73bb6647db" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:13:13.951613 32641 raft_consensus.cc:2454] T efd50e7bc6f04e43bba12e73bb6647db P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:13:13.951686 3055 tablet_bootstrap.cc:492] T eb0f4fe6b8684f329725ac2f7d5428f2 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:13:13.952117 32709 leader_election.cc:302] T efd50e7bc6f04e43bba12e73bb6647db P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, 9a4d92bc25374fb3960fde7d10be309d; no voters:
I20240503 02:13:13.952433 2931 raft_consensus.cc:2781] T efd50e7bc6f04e43bba12e73bb6647db P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:13.952533 2931 raft_consensus.cc:482] T efd50e7bc6f04e43bba12e73bb6647db P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:13.952420 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd50e7bc6f04e43bba12e73bb6647db" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:13:13.952654 2931 raft_consensus.cc:3037] T efd50e7bc6f04e43bba12e73bb6647db P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:13.952842 437 raft_consensus.cc:2454] T efd50e7bc6f04e43bba12e73bb6647db P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:13:13.953137 3055 tablet_bootstrap.cc:654] T eb0f4fe6b8684f329725ac2f7d5428f2 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:13.955469 2931 raft_consensus.cc:504] T efd50e7bc6f04e43bba12e73bb6647db P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:13.956126 2931 leader_election.cc:288] T efd50e7bc6f04e43bba12e73bb6647db P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Requested vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:13.956612 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd50e7bc6f04e43bba12e73bb6647db" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d"
I20240503 02:13:13.956885 32641 raft_consensus.cc:3037] T efd50e7bc6f04e43bba12e73bb6647db P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:13.964365 32641 raft_consensus.cc:2454] T efd50e7bc6f04e43bba12e73bb6647db P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:13:13.968626 32709 leader_election.cc:302] T efd50e7bc6f04e43bba12e73bb6647db P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, 9a4d92bc25374fb3960fde7d10be309d; no voters:
I20240503 02:13:13.969053 2931 raft_consensus.cc:2781] T efd50e7bc6f04e43bba12e73bb6647db P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:13.969224 2931 raft_consensus.cc:686] T efd50e7bc6f04e43bba12e73bb6647db P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Becoming Leader. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Running, Role: LEADER
I20240503 02:13:13.969519 2931 consensus_queue.cc:237] T efd50e7bc6f04e43bba12e73bb6647db P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:13.976730 3057 tablet_bootstrap.cc:492] T eb0f4fe6b8684f329725ac2f7d5428f2 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:13.979090 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efd50e7bc6f04e43bba12e73bb6647db" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:13:13.979347 437 raft_consensus.cc:3037] T efd50e7bc6f04e43bba12e73bb6647db P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:13.981395 3055 tablet_bootstrap.cc:492] T eb0f4fe6b8684f329725ac2f7d5428f2 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:13:13.984113 3055 ts_tablet_manager.cc:1367] T eb0f4fe6b8684f329725ac2f7d5428f2 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.003s
I20240503 02:13:13.984532 437 raft_consensus.cc:2454] T efd50e7bc6f04e43bba12e73bb6647db P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:13:13.986369 3057 tablet_bootstrap.cc:654] T eb0f4fe6b8684f329725ac2f7d5428f2 P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:13.987835 3055 raft_consensus.cc:348] T eb0f4fe6b8684f329725ac2f7d5428f2 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:13.991393 3055 raft_consensus.cc:374] T eb0f4fe6b8684f329725ac2f7d5428f2 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:13.991556 3055 raft_consensus.cc:729] T eb0f4fe6b8684f329725ac2f7d5428f2 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:13:13.992134 3055 consensus_queue.cc:260] T eb0f4fe6b8684f329725ac2f7d5428f2 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:13.994699 3054 tablet_bootstrap.cc:492] T 7a6f23bb933949e9868139608456130f P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:13:13.999981 3055 ts_tablet_manager.cc:1398] T eb0f4fe6b8684f329725ac2f7d5428f2 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.013s user 0.000s sys 0.001s
I20240503 02:13:14.011864 3054 tablet_bootstrap.cc:654] T 7a6f23bb933949e9868139608456130f P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:14.013516 3058 tablet_bootstrap.cc:492] T e165c71f8b6645a6936b58b2a8da1842 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:13:14.015828 3058 tablet_bootstrap.cc:654] T e165c71f8b6645a6936b58b2a8da1842 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:14.019474 29730 catalog_manager.cc:5484] T efd50e7bc6f04e43bba12e73bb6647db P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6). New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:14.023689 2755 raft_consensus.cc:482] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:14.023864 2755 raft_consensus.cc:504] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:14.024526 2755 leader_election.cc:288] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:13:14.025346 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a6f23bb933949e9868139608456130f" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:13:14.025589 32641 raft_consensus.cc:2454] T 7a6f23bb933949e9868139608456130f P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:13:14.025663 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a6f23bb933949e9868139608456130f" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:13:14.026173 29809 leader_election.cc:302] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, 9a4d92bc25374fb3960fde7d10be309d; no voters:
W20240503 02:13:14.026446 29809 leader_election.cc:341] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:14.027318 2755 raft_consensus.cc:2781] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:14.027395 2755 raft_consensus.cc:482] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:14.027447 2755 raft_consensus.cc:3037] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:14.031085 2755 raft_consensus.cc:504] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:14.032307 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a6f23bb933949e9868139608456130f" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
W20240503 02:13:14.032773 29809 leader_election.cc:341] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:14.032838 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a6f23bb933949e9868139608456130f" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d"
I20240503 02:13:14.033196 32641 raft_consensus.cc:3037] T 7a6f23bb933949e9868139608456130f P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:14.036748 32641 raft_consensus.cc:2454] T 7a6f23bb933949e9868139608456130f P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:13:14.037240 29809 leader_election.cc:302] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16440e365430405fb10366ab70e2b429, 9a4d92bc25374fb3960fde7d10be309d; no voters: 155735c7f8b14997ac4c770aee2a9401
I20240503 02:13:14.037591 2947 raft_consensus.cc:2781] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:14.037714 2947 raft_consensus.cc:686] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Becoming Leader. State: Replica: 16440e365430405fb10366ab70e2b429, State: Running, Role: LEADER
I20240503 02:13:14.038025 2947 consensus_queue.cc:237] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:14.040689 2755 leader_election.cc:288] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Requested vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:13:14.063069 3057 tablet_bootstrap.cc:492] T eb0f4fe6b8684f329725ac2f7d5428f2 P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:14.063328 3057 ts_tablet_manager.cc:1367] T eb0f4fe6b8684f329725ac2f7d5428f2 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.087s user 0.003s sys 0.014s
I20240503 02:13:14.064142 3057 raft_consensus.cc:348] T eb0f4fe6b8684f329725ac2f7d5428f2 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:14.064455 3057 raft_consensus.cc:374] T eb0f4fe6b8684f329725ac2f7d5428f2 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:14.064579 3057 raft_consensus.cc:729] T eb0f4fe6b8684f329725ac2f7d5428f2 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:14.064872 3057 consensus_queue.cc:260] T eb0f4fe6b8684f329725ac2f7d5428f2 P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:14.065532 3057 ts_tablet_manager.cc:1398] T eb0f4fe6b8684f329725ac2f7d5428f2 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:14.066025 3057 tablet_bootstrap.cc:492] T e165c71f8b6645a6936b58b2a8da1842 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:14.067615 3057 tablet_bootstrap.cc:654] T e165c71f8b6645a6936b58b2a8da1842 P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:14.073233 29725 catalog_manager.cc:5484] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16440e365430405fb10366ab70e2b429 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:14.078469 3054 tablet_bootstrap.cc:492] T 7a6f23bb933949e9868139608456130f P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:13:14.078670 3054 ts_tablet_manager.cc:1367] T 7a6f23bb933949e9868139608456130f P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.084s user 0.002s sys 0.021s
I20240503 02:13:14.079355 3054 raft_consensus.cc:348] T 7a6f23bb933949e9868139608456130f P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:14.079610 3054 raft_consensus.cc:374] T 7a6f23bb933949e9868139608456130f P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:14.079707 3054 raft_consensus.cc:729] T 7a6f23bb933949e9868139608456130f P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:13:14.079993 3054 consensus_queue.cc:260] T 7a6f23bb933949e9868139608456130f P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:14.080670 3054 ts_tablet_manager.cc:1398] T 7a6f23bb933949e9868139608456130f P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:13:14.084848 3058 tablet_bootstrap.cc:492] T e165c71f8b6645a6936b58b2a8da1842 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:13:14.085005 3058 ts_tablet_manager.cc:1367] T e165c71f8b6645a6936b58b2a8da1842 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.072s user 0.004s sys 0.000s
I20240503 02:13:14.097620 3058 raft_consensus.cc:348] T e165c71f8b6645a6936b58b2a8da1842 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:14.097857 3058 raft_consensus.cc:374] T e165c71f8b6645a6936b58b2a8da1842 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:14.097944 3058 raft_consensus.cc:729] T e165c71f8b6645a6936b58b2a8da1842 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:13:14.098197 3058 consensus_queue.cc:260] T e165c71f8b6645a6936b58b2a8da1842 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:14.100714 3058 ts_tablet_manager.cc:1398] T e165c71f8b6645a6936b58b2a8da1842 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240503 02:13:14.101514 2937 raft_consensus.cc:482] T 7a6f23bb933949e9868139608456130f P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:14.101641 2937 raft_consensus.cc:504] T 7a6f23bb933949e9868139608456130f P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:14.102422 2937 leader_election.cc:288] T 7a6f23bb933949e9868139608456130f P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:13:14.103381 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a6f23bb933949e9868139608456130f" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:13:14.103643 32641 raft_consensus.cc:2379] T 7a6f23bb933949e9868139608456130f P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 155735c7f8b14997ac4c770aee2a9401 in current term 1: Already voted for candidate 16440e365430405fb10366ab70e2b429 in this term.
I20240503 02:13:14.104831 29872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a6f23bb933949e9868139608456130f" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:13:14.105458 29942 leader_election.cc:302] T 7a6f23bb933949e9868139608456130f P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401; no voters: 16440e365430405fb10366ab70e2b429, 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:13:14.106101 2942 raft_consensus.cc:3037] T 7a6f23bb933949e9868139608456130f P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:14.108731 2942 raft_consensus.cc:2726] T 7a6f23bb933949e9868139608456130f P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:14.112494 3057 tablet_bootstrap.cc:492] T e165c71f8b6645a6936b58b2a8da1842 P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:14.112746 3057 ts_tablet_manager.cc:1367] T e165c71f8b6645a6936b58b2a8da1842 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.047s user 0.006s sys 0.022s
I20240503 02:13:14.114266 3057 raft_consensus.cc:348] T e165c71f8b6645a6936b58b2a8da1842 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:14.114559 3057 raft_consensus.cc:374] T e165c71f8b6645a6936b58b2a8da1842 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:14.114631 3057 raft_consensus.cc:729] T e165c71f8b6645a6936b58b2a8da1842 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:14.114876 3057 consensus_queue.cc:260] T e165c71f8b6645a6936b58b2a8da1842 P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:14.118482 3057 ts_tablet_manager.cc:1398] T e165c71f8b6645a6936b58b2a8da1842 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:13:14.157208 2931 raft_consensus.cc:482] T e165c71f8b6645a6936b58b2a8da1842 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:14.157383 2931 raft_consensus.cc:504] T e165c71f8b6645a6936b58b2a8da1842 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:14.157995 2931 leader_election.cc:288] T e165c71f8b6645a6936b58b2a8da1842 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:13:14.158708 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e165c71f8b6645a6936b58b2a8da1842" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:13:14.158686 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e165c71f8b6645a6936b58b2a8da1842" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:14.158912 30134 raft_consensus.cc:2454] T e165c71f8b6645a6936b58b2a8da1842 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:13:14.158933 567 raft_consensus.cc:2454] T e165c71f8b6645a6936b58b2a8da1842 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:13:14.159370 32708 leader_election.cc:302] T e165c71f8b6645a6936b58b2a8da1842 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:13:14.159626 2931 raft_consensus.cc:2781] T e165c71f8b6645a6936b58b2a8da1842 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:14.159754 2931 raft_consensus.cc:482] T e165c71f8b6645a6936b58b2a8da1842 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:14.159852 2931 raft_consensus.cc:3037] T e165c71f8b6645a6936b58b2a8da1842 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:14.161687 30265 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7527cc5931e24fa180efedbf81438efd"
dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40742
I20240503 02:13:14.161922 30265 raft_consensus.cc:595] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:13:14.162451 2931 raft_consensus.cc:504] T e165c71f8b6645a6936b58b2a8da1842 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:14.163081 2931 leader_election.cc:288] T e165c71f8b6645a6936b58b2a8da1842 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Requested vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:13:14.163628 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e165c71f8b6645a6936b58b2a8da1842" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:13:14.163854 567 raft_consensus.cc:3037] T e165c71f8b6645a6936b58b2a8da1842 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:14.163902 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e165c71f8b6645a6936b58b2a8da1842" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:13:14.164126 30134 raft_consensus.cc:3037] T e165c71f8b6645a6936b58b2a8da1842 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:14.167279 567 raft_consensus.cc:2454] T e165c71f8b6645a6936b58b2a8da1842 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:13:14.169425 32708 leader_election.cc:302] T e165c71f8b6645a6936b58b2a8da1842 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:14.167392 30134 raft_consensus.cc:2454] T e165c71f8b6645a6936b58b2a8da1842 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:13:14.170006 2931 raft_consensus.cc:2781] T e165c71f8b6645a6936b58b2a8da1842 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:14.170953 2931 raft_consensus.cc:686] T e165c71f8b6645a6936b58b2a8da1842 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Becoming Leader. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Running, Role: LEADER
I20240503 02:13:14.171422 2931 consensus_queue.cc:237] T e165c71f8b6645a6936b58b2a8da1842 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:14.178697 29730 catalog_manager.cc:5484] T e165c71f8b6645a6936b58b2a8da1842 P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6). New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:14.183112 29730 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:13:14.192059 437 raft_consensus.cc:1261] T efd50e7bc6f04e43bba12e73bb6647db P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:14.192116 32641 raft_consensus.cc:1261] T efd50e7bc6f04e43bba12e73bb6647db P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:14.192653 2787 consensus_queue.cc:1035] T efd50e7bc6f04e43bba12e73bb6647db P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:14.192988 3076 consensus_queue.cc:1035] T efd50e7bc6f04e43bba12e73bb6647db P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:14.199193 29730 catalog_manager.cc:5576] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241) does not have the latest schema for tablet efd50e7bc6f04e43bba12e73bb6647db (table rebalancer_dynamic_table [id=c388300a2a094bd4b9811c52202ec490]). Expected version 1 got 0
I20240503 02:13:14.200599 3068 raft_consensus.cc:482] T eb0f4fe6b8684f329725ac2f7d5428f2 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:14.200778 3068 raft_consensus.cc:504] T eb0f4fe6b8684f329725ac2f7d5428f2 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:14.201800 3068 leader_election.cc:288] T eb0f4fe6b8684f329725ac2f7d5428f2 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:14.202242 29872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb0f4fe6b8684f329725ac2f7d5428f2" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:13:14.202580 29872 raft_consensus.cc:2454] T eb0f4fe6b8684f329725ac2f7d5428f2 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 0.
I20240503 02:13:14.203337 504 leader_election.cc:302] T eb0f4fe6b8684f329725ac2f7d5428f2 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:14.203883 3068 raft_consensus.cc:2781] T eb0f4fe6b8684f329725ac2f7d5428f2 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:14.204046 3068 raft_consensus.cc:482] T eb0f4fe6b8684f329725ac2f7d5428f2 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:14.204157 3068 raft_consensus.cc:3037] T eb0f4fe6b8684f329725ac2f7d5428f2 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:14.207892 3068 raft_consensus.cc:504] T eb0f4fe6b8684f329725ac2f7d5428f2 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:14.208731 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb0f4fe6b8684f329725ac2f7d5428f2" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:13:14.209081 307 raft_consensus.cc:2454] T eb0f4fe6b8684f329725ac2f7d5428f2 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 0.
I20240503 02:13:14.214165 29730 catalog_manager.cc:5576] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595) does not have the latest schema for tablet efd50e7bc6f04e43bba12e73bb6647db (table rebalancer_dynamic_table [id=c388300a2a094bd4b9811c52202ec490]). Expected version 1 got 0
I20240503 02:13:14.218509 29872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb0f4fe6b8684f329725ac2f7d5428f2" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:13:14.218777 29872 raft_consensus.cc:3037] T eb0f4fe6b8684f329725ac2f7d5428f2 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:14.224567 3090 tablet.cc:1722] T efd50e7bc6f04e43bba12e73bb6647db P 9a4d92bc25374fb3960fde7d10be309d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:14.225104 3088 tablet.cc:1722] T efd50e7bc6f04e43bba12e73bb6647db P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:14.225210 29872 raft_consensus.cc:2454] T eb0f4fe6b8684f329725ac2f7d5428f2 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 1.
I20240503 02:13:14.225653 504 leader_election.cc:302] T eb0f4fe6b8684f329725ac2f7d5428f2 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:14.225775 3088 tablet.cc:1620] T efd50e7bc6f04e43bba12e73bb6647db P 8a666e798fe8477d8dd1e51b385d5ead: MemRowSet was empty: no flush needed.
I20240503 02:13:14.226001 2946 raft_consensus.cc:2781] T eb0f4fe6b8684f329725ac2f7d5428f2 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:14.227231 3089 tablet.cc:1722] T efd50e7bc6f04e43bba12e73bb6647db P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:14.227855 3089 tablet.cc:1620] T efd50e7bc6f04e43bba12e73bb6647db P fbbc2be6382e47908ecb7c68c367bd26: MemRowSet was empty: no flush needed.
I20240503 02:13:14.228049 29730 catalog_manager.cc:5576] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241) does not have the latest schema for tablet efd50e7bc6f04e43bba12e73bb6647db (table rebalancer_dynamic_table [id=c388300a2a094bd4b9811c52202ec490]). Expected version 1 got 0
I20240503 02:13:14.228672 3090 tablet.cc:1620] T efd50e7bc6f04e43bba12e73bb6647db P 9a4d92bc25374fb3960fde7d10be309d: MemRowSet was empty: no flush needed.
I20240503 02:13:14.230048 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb0f4fe6b8684f329725ac2f7d5428f2" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:13:14.230324 307 raft_consensus.cc:3037] T eb0f4fe6b8684f329725ac2f7d5428f2 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:14.231842 29730 catalog_manager.cc:5576] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595) does not have the latest schema for tablet efd50e7bc6f04e43bba12e73bb6647db (table rebalancer_dynamic_table [id=c388300a2a094bd4b9811c52202ec490]). Expected version 1 got 0
I20240503 02:13:14.233137 2324 raft_consensus.cc:979] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012: : Instructing follower f575ce92c0224eeb80b9fae5f6561fbe to start an election
I20240503 02:13:14.233276 2324 raft_consensus.cc:1067] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Signalling peer f575ce92c0224eeb80b9fae5f6561fbe to start an election
I20240503 02:13:14.234333 567 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7527cc5931e24fa180efedbf81438efd"
dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
from {username='slave'} at 127.22.108.4:45935
I20240503 02:13:14.234584 567 raft_consensus.cc:482] T 7527cc5931e24fa180efedbf81438efd P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:14.234710 567 raft_consensus.cc:3037] T 7527cc5931e24fa180efedbf81438efd P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:14.237862 307 raft_consensus.cc:2454] T eb0f4fe6b8684f329725ac2f7d5428f2 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 1.
I20240503 02:13:14.240990 29730 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=c388300a2a094bd4b9811c52202ec490] alter complete (version 1)
I20240503 02:13:14.242270 567 raft_consensus.cc:504] T 7527cc5931e24fa180efedbf81438efd P f575ce92c0224eeb80b9fae5f6561fbe [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:14.242765 3068 leader_election.cc:288] T eb0f4fe6b8684f329725ac2f7d5428f2 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 election: Requested vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:14.245051 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7527cc5931e24fa180efedbf81438efd" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:13:14.245321 30134 raft_consensus.cc:3037] T 7527cc5931e24fa180efedbf81438efd P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:14.247727 29872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7527cc5931e24fa180efedbf81438efd" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:13:14.247929 29872 raft_consensus.cc:3037] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:14.248219 30134 raft_consensus.cc:2454] T 7527cc5931e24fa180efedbf81438efd P a9094240860e43bfb129193f3bfd5be6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 2.
I20240503 02:13:14.249642 567 leader_election.cc:288] T 7527cc5931e24fa180efedbf81438efd P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 2 election: Requested vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:13:14.252509 29872 raft_consensus.cc:2454] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 2.
I20240503 02:13:14.255438 504 leader_election.cc:302] T 7527cc5931e24fa180efedbf81438efd P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, a9094240860e43bfb129193f3bfd5be6, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:14.255759 3068 raft_consensus.cc:2781] T 7527cc5931e24fa180efedbf81438efd P f575ce92c0224eeb80b9fae5f6561fbe [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:14.255893 3068 raft_consensus.cc:686] T 7527cc5931e24fa180efedbf81438efd P f575ce92c0224eeb80b9fae5f6561fbe [term 2 LEADER]: Becoming Leader. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Running, Role: LEADER
I20240503 02:13:14.256150 3068 consensus_queue.cc:237] T 7527cc5931e24fa180efedbf81438efd P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:14.259459 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7527cc5931e24fa180efedbf81438efd" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:13:14.259688 30265 raft_consensus.cc:3032] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:14.259644 2946 raft_consensus.cc:686] T eb0f4fe6b8684f329725ac2f7d5428f2 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 LEADER]: Becoming Leader. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Running, Role: LEADER
I20240503 02:13:14.259768 30265 raft_consensus.cc:729] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Running, Role: LEADER
I20240503 02:13:14.260193 2946 consensus_queue.cc:237] T eb0f4fe6b8684f329725ac2f7d5428f2 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:14.261945 30265 consensus_queue.cc:260] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:14.262339 30265 raft_consensus.cc:3037] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:14.264503 30265 raft_consensus.cc:2454] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 2.
I20240503 02:13:14.269922 29730 catalog_manager.cc:5484] T eb0f4fe6b8684f329725ac2f7d5428f2 P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: term changed from 0 to 1, leader changed from <none> to f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8). New cstate: current_term: 1 leader_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:14.276444 29730 catalog_manager.cc:5484] T 7527cc5931e24fa180efedbf81438efd P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: term changed from 1 to 2, leader changed from 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) to f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8). New cstate: current_term: 2 leader_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:14.348641 29730 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:13:14.360571 3088 tablet.cc:1722] T efd50e7bc6f04e43bba12e73bb6647db P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:14.361143 3088 tablet.cc:1620] T efd50e7bc6f04e43bba12e73bb6647db P 8a666e798fe8477d8dd1e51b385d5ead: MemRowSet was empty: no flush needed.
I20240503 02:13:14.365188 3090 tablet.cc:1722] T efd50e7bc6f04e43bba12e73bb6647db P 9a4d92bc25374fb3960fde7d10be309d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:14.365742 3090 tablet.cc:1620] T efd50e7bc6f04e43bba12e73bb6647db P 9a4d92bc25374fb3960fde7d10be309d: MemRowSet was empty: no flush needed.
I20240503 02:13:14.365779 3089 tablet.cc:1722] T efd50e7bc6f04e43bba12e73bb6647db P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:14.366246 3089 tablet.cc:1620] T efd50e7bc6f04e43bba12e73bb6647db P fbbc2be6382e47908ecb7c68c367bd26: MemRowSet was empty: no flush needed.
I20240503 02:13:14.377815 29732 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=c388300a2a094bd4b9811c52202ec490] alter complete (version 2)
I20240503 02:13:14.440140 2755 consensus_queue.cc:1035] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:14.450031 2755 consensus_queue.cc:1035] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:14.482100 29734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:13:14.496587 3088 tablet.cc:1722] T efd50e7bc6f04e43bba12e73bb6647db P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:14.497860 3089 tablet.cc:1722] T efd50e7bc6f04e43bba12e73bb6647db P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:14.501863 3090 tablet.cc:1722] T efd50e7bc6f04e43bba12e73bb6647db P 9a4d92bc25374fb3960fde7d10be309d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:14.510567 29734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table_new_name [id=c388300a2a094bd4b9811c52202ec490] alter complete (version 3)
I20240503 02:13:14.569736 29853 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:13:14.574234 29984 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:13:14.575387 30245 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:14.597332 30116 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:14.613723 29734 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:14.613972 29734 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:14.617651 29734 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet efd50e7bc6f04e43bba12e73bb6647db
I20240503 02:13:14.619594 32754 tablet_service.cc:1514] Processing DeleteTablet for tablet efd50e7bc6f04e43bba12e73bb6647db with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:14 UTC) from {username='slave'} at 127.0.0.1:46360
I20240503 02:13:14.619585 416 tablet_service.cc:1514] Processing DeleteTablet for tablet efd50e7bc6f04e43bba12e73bb6647db with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:14 UTC) from {username='slave'} at 127.0.0.1:50814
I20240503 02:13:14.620204 32620 tablet_service.cc:1514] Processing DeleteTablet for tablet efd50e7bc6f04e43bba12e73bb6647db with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:14 UTC) from {username='slave'} at 127.0.0.1:38200
I20240503 02:13:14.633416 32755 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:13:14.643107 32619 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:14.651357 30135 raft_consensus.cc:1261] T 7527cc5931e24fa180efedbf81438efd P a9094240860e43bfb129193f3bfd5be6 [term 2 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:13:14.651968 3068 consensus_queue.cc:1035] T 7527cc5931e24fa180efedbf81438efd P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:14.655081 3148 tablet_replica.cc:330] T efd50e7bc6f04e43bba12e73bb6647db P 8a666e798fe8477d8dd1e51b385d5ead: stopping tablet replica
I20240503 02:13:14.655341 3148 raft_consensus.cc:2229] T efd50e7bc6f04e43bba12e73bb6647db P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:14.655781 3148 raft_consensus.cc:2258] T efd50e7bc6f04e43bba12e73bb6647db P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:14.659258 3147 tablet_replica.cc:330] T efd50e7bc6f04e43bba12e73bb6647db P fbbc2be6382e47908ecb7c68c367bd26: stopping tablet replica
I20240503 02:13:14.659595 3147 raft_consensus.cc:2229] T efd50e7bc6f04e43bba12e73bb6647db P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:14.659873 3147 raft_consensus.cc:2258] T efd50e7bc6f04e43bba12e73bb6647db P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:14.664651 30266 raft_consensus.cc:1261] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:13:14.665300 3068 consensus_queue.cc:1035] T 7527cc5931e24fa180efedbf81438efd P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:14.666635 3147 ts_tablet_manager.cc:1875] T efd50e7bc6f04e43bba12e73bb6647db P fbbc2be6382e47908ecb7c68c367bd26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:14.677955 3149 tablet_replica.cc:330] T efd50e7bc6f04e43bba12e73bb6647db P 9a4d92bc25374fb3960fde7d10be309d: stopping tablet replica
I20240503 02:13:14.678216 3149 raft_consensus.cc:2229] T efd50e7bc6f04e43bba12e73bb6647db P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:14.678946 3149 raft_consensus.cc:2258] T efd50e7bc6f04e43bba12e73bb6647db P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:14.680735 3148 ts_tablet_manager.cc:1875] T efd50e7bc6f04e43bba12e73bb6647db P 8a666e798fe8477d8dd1e51b385d5ead: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:14.681331 3149 ts_tablet_manager.cc:1875] T efd50e7bc6f04e43bba12e73bb6647db P 9a4d92bc25374fb3960fde7d10be309d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:14.688159 3147 ts_tablet_manager.cc:1888] T efd50e7bc6f04e43bba12e73bb6647db P fbbc2be6382e47908ecb7c68c367bd26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:14.695474 29872 raft_consensus.cc:1261] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 [term 2 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:13:14.696029 3147 log.cc:1196] T efd50e7bc6f04e43bba12e73bb6647db P fbbc2be6382e47908ecb7c68c367bd26: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal/wals/efd50e7bc6f04e43bba12e73bb6647db
I20240503 02:13:14.696170 2946 consensus_queue.cc:1035] T 7527cc5931e24fa180efedbf81438efd P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:14.697393 3147 ts_tablet_manager.cc:1909] T efd50e7bc6f04e43bba12e73bb6647db P fbbc2be6382e47908ecb7c68c367bd26: Deleting consensus metadata
I20240503 02:13:14.698638 29719 catalog_manager.cc:4830] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): tablet efd50e7bc6f04e43bba12e73bb6647db (table rebalancer_dynamic_table_new_name [id=c388300a2a094bd4b9811c52202ec490]) successfully deleted
I20240503 02:13:14.719220 3149 ts_tablet_manager.cc:1888] T efd50e7bc6f04e43bba12e73bb6647db P 9a4d92bc25374fb3960fde7d10be309d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:14.721731 3149 log.cc:1196] T efd50e7bc6f04e43bba12e73bb6647db P 9a4d92bc25374fb3960fde7d10be309d: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal/wals/efd50e7bc6f04e43bba12e73bb6647db
I20240503 02:13:14.722508 29734 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:13:14.725826 3149 ts_tablet_manager.cc:1909] T efd50e7bc6f04e43bba12e73bb6647db P 9a4d92bc25374fb3960fde7d10be309d: Deleting consensus metadata
I20240503 02:13:14.727409 29719 catalog_manager.cc:4830] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): tablet efd50e7bc6f04e43bba12e73bb6647db (table rebalancer_dynamic_table_new_name [id=c388300a2a094bd4b9811c52202ec490]) successfully deleted
I20240503 02:13:14.732087 416 tablet_service.cc:1467] Processing CreateTablet for tablet 76c9c41a3669428592ed5674a72b7ab3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=54927021e06c431686f6a9e7200dfd2d]), partition=
I20240503 02:13:14.732194 29984 tablet_service.cc:1467] Processing CreateTablet for tablet 76c9c41a3669428592ed5674a72b7ab3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=54927021e06c431686f6a9e7200dfd2d]), partition=
I20240503 02:13:14.732465 416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76c9c41a3669428592ed5674a72b7ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:14.732597 29984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76c9c41a3669428592ed5674a72b7ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:14.737963 32754 tablet_service.cc:1467] Processing CreateTablet for tablet 76c9c41a3669428592ed5674a72b7ab3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=54927021e06c431686f6a9e7200dfd2d]), partition=
I20240503 02:13:14.743614 2787 consensus_queue.cc:1035] T e165c71f8b6645a6936b58b2a8da1842 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:14.748822 32754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76c9c41a3669428592ed5674a72b7ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:14.759132 3148 ts_tablet_manager.cc:1888] T efd50e7bc6f04e43bba12e73bb6647db P 8a666e798fe8477d8dd1e51b385d5ead: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:14.759383 3148 log.cc:1196] T efd50e7bc6f04e43bba12e73bb6647db P 8a666e798fe8477d8dd1e51b385d5ead: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal/wals/efd50e7bc6f04e43bba12e73bb6647db
I20240503 02:13:14.760084 3148 ts_tablet_manager.cc:1909] T efd50e7bc6f04e43bba12e73bb6647db P 8a666e798fe8477d8dd1e51b385d5ead: Deleting consensus metadata
I20240503 02:13:14.761240 29719 catalog_manager.cc:4830] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): tablet efd50e7bc6f04e43bba12e73bb6647db (table rebalancer_dynamic_table_new_name [id=c388300a2a094bd4b9811c52202ec490]) successfully deleted
I20240503 02:13:14.768456 2787 consensus_queue.cc:1035] T e165c71f8b6645a6936b58b2a8da1842 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:14.777904 3162 tablet_bootstrap.cc:492] T 76c9c41a3669428592ed5674a72b7ab3 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:14.795711 3162 tablet_bootstrap.cc:654] T 76c9c41a3669428592ed5674a72b7ab3 P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:14.812696 29734 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_extra_table_36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:14.813164 417 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:13:14.815560 3159 tablet_bootstrap.cc:492] T 76c9c41a3669428592ed5674a72b7ab3 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:13:14.824875 547 tablet_service.cc:1467] Processing CreateTablet for tablet 4563386631774e74a4b12fc71714c1a0 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=9f19e29c56ab440eb34df78f91d5bd29]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:14.825414 547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4563386631774e74a4b12fc71714c1a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:14.825423 30245 tablet_service.cc:1467] Processing CreateTablet for tablet 3f3ec15b1ec043d1a8f252f42a584f63 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=9f19e29c56ab440eb34df78f91d5bd29]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:14.825467 32620 tablet_service.cc:1467] Processing CreateTablet for tablet 3f3ec15b1ec043d1a8f252f42a584f63 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=9f19e29c56ab440eb34df78f91d5bd29]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:14.825790 30245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f3ec15b1ec043d1a8f252f42a584f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:14.825930 32620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f3ec15b1ec043d1a8f252f42a584f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:14.830018 2946 consensus_queue.cc:1035] T eb0f4fe6b8684f329725ac2f7d5428f2 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:14.830029 29986 tablet_service.cc:1467] Processing CreateTablet for tablet 4563386631774e74a4b12fc71714c1a0 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=9f19e29c56ab440eb34df78f91d5bd29]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:14.830431 29986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4563386631774e74a4b12fc71714c1a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:14.830403 417 tablet_service.cc:1467] Processing CreateTablet for tablet ec3891bfc4014b63afef6c259de00ca7 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=9f19e29c56ab440eb34df78f91d5bd29]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:14.830837 417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec3891bfc4014b63afef6c259de00ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:14.833935 32621 tablet_service.cc:1467] Processing CreateTablet for tablet ec3891bfc4014b63afef6c259de00ca7 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=9f19e29c56ab440eb34df78f91d5bd29]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:14.834337 32621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec3891bfc4014b63afef6c259de00ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:14.835656 32755 tablet_service.cc:1467] Processing CreateTablet for tablet ec3891bfc4014b63afef6c259de00ca7 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=9f19e29c56ab440eb34df78f91d5bd29]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:14.833889 32619 tablet_service.cc:1467] Processing CreateTablet for tablet 4563386631774e74a4b12fc71714c1a0 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=9f19e29c56ab440eb34df78f91d5bd29]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:14.836040 32755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec3891bfc4014b63afef6c259de00ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:14.836854 32754 tablet_service.cc:1467] Processing CreateTablet for tablet 3f3ec15b1ec043d1a8f252f42a584f63 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=9f19e29c56ab440eb34df78f91d5bd29]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:14.837692 32754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f3ec15b1ec043d1a8f252f42a584f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:14.838667 32619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4563386631774e74a4b12fc71714c1a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:14.847505 3162 tablet_bootstrap.cc:492] T 76c9c41a3669428592ed5674a72b7ab3 P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:14.847718 3162 ts_tablet_manager.cc:1367] T 76c9c41a3669428592ed5674a72b7ab3 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.070s user 0.000s sys 0.005s
I20240503 02:13:14.848601 3162 raft_consensus.cc:348] T 76c9c41a3669428592ed5674a72b7ab3 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:14.848977 3162 raft_consensus.cc:374] T 76c9c41a3669428592ed5674a72b7ab3 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:14.849146 3162 raft_consensus.cc:729] T 76c9c41a3669428592ed5674a72b7ab3 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:14.849473 3162 consensus_queue.cc:260] T 76c9c41a3669428592ed5674a72b7ab3 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:14.850225 3162 ts_tablet_manager.cc:1398] T 76c9c41a3669428592ed5674a72b7ab3 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:14.850535 3162 tablet_bootstrap.cc:492] T ec3891bfc4014b63afef6c259de00ca7 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:14.853718 3162 tablet_bootstrap.cc:654] T ec3891bfc4014b63afef6c259de00ca7 P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:14.858127 3159 tablet_bootstrap.cc:654] T 76c9c41a3669428592ed5674a72b7ab3 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:14.860703 2946 consensus_queue.cc:1035] T eb0f4fe6b8684f329725ac2f7d5428f2 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:14.865913 3161 tablet_bootstrap.cc:492] T 76c9c41a3669428592ed5674a72b7ab3 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:14.872501 3168 tablet_bootstrap.cc:492] T 3f3ec15b1ec043d1a8f252f42a584f63 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:13:14.873598 3167 tablet_bootstrap.cc:492] T ec3891bfc4014b63afef6c259de00ca7 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:14.873805 3159 tablet_bootstrap.cc:492] T 76c9c41a3669428592ed5674a72b7ab3 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:13:14.874341 3159 ts_tablet_manager.cc:1367] T 76c9c41a3669428592ed5674a72b7ab3 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.059s user 0.005s sys 0.000s
I20240503 02:13:14.876904 3168 tablet_bootstrap.cc:654] T 3f3ec15b1ec043d1a8f252f42a584f63 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:14.878060 3159 raft_consensus.cc:348] T 76c9c41a3669428592ed5674a72b7ab3 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:14.878968 3162 tablet_bootstrap.cc:492] T ec3891bfc4014b63afef6c259de00ca7 P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:14.878979 3166 tablet_bootstrap.cc:492] T 4563386631774e74a4b12fc71714c1a0 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:14.879215 3162 ts_tablet_manager.cc:1367] T ec3891bfc4014b63afef6c259de00ca7 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.001s
I20240503 02:13:14.880174 3162 raft_consensus.cc:348] T ec3891bfc4014b63afef6c259de00ca7 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:14.880450 3162 raft_consensus.cc:374] T ec3891bfc4014b63afef6c259de00ca7 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:14.880565 3162 raft_consensus.cc:729] T ec3891bfc4014b63afef6c259de00ca7 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:14.880913 3162 consensus_queue.cc:260] T ec3891bfc4014b63afef6c259de00ca7 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:14.881621 3162 ts_tablet_manager.cc:1398] T ec3891bfc4014b63afef6c259de00ca7 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:14.881969 3166 tablet_bootstrap.cc:654] T 4563386631774e74a4b12fc71714c1a0 P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:14.882184 3159 raft_consensus.cc:374] T 76c9c41a3669428592ed5674a72b7ab3 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:14.882304 3159 raft_consensus.cc:729] T 76c9c41a3669428592ed5674a72b7ab3 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:13:14.882835 3159 consensus_queue.cc:260] T 76c9c41a3669428592ed5674a72b7ab3 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:14.884581 3159 ts_tablet_manager.cc:1398] T 76c9c41a3669428592ed5674a72b7ab3 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:13:14.884907 3159 tablet_bootstrap.cc:492] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:13:14.886262 2787 raft_consensus.cc:482] T 76c9c41a3669428592ed5674a72b7ab3 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:14.886394 2787 raft_consensus.cc:504] T 76c9c41a3669428592ed5674a72b7ab3 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:14.887014 2787 leader_election.cc:288] T 76c9c41a3669428592ed5674a72b7ab3 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:14.888963 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76c9c41a3669428592ed5674a72b7ab3" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:13:14.889178 30004 raft_consensus.cc:2454] T 76c9c41a3669428592ed5674a72b7ab3 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:13:14.889720 32709 leader_election.cc:302] T 76c9c41a3669428592ed5674a72b7ab3 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 8a666e798fe8477d8dd1e51b385d5ead; no voters:
I20240503 02:13:14.890082 2787 raft_consensus.cc:2781] T 76c9c41a3669428592ed5674a72b7ab3 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:14.890188 2787 raft_consensus.cc:482] T 76c9c41a3669428592ed5674a72b7ab3 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:14.890282 2787 raft_consensus.cc:3037] T 76c9c41a3669428592ed5674a72b7ab3 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:14.891657 435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76c9c41a3669428592ed5674a72b7ab3" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
W20240503 02:13:14.892295 32709 leader_election.cc:341] T 76c9c41a3669428592ed5674a72b7ab3 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:14.893081 3161 tablet_bootstrap.cc:654] T 76c9c41a3669428592ed5674a72b7ab3 P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:14.894201 2787 raft_consensus.cc:504] T 76c9c41a3669428592ed5674a72b7ab3 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:14.894944 3167 tablet_bootstrap.cc:654] T ec3891bfc4014b63afef6c259de00ca7 P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:14.895430 2787 leader_election.cc:288] T 76c9c41a3669428592ed5674a72b7ab3 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Requested vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:14.896343 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76c9c41a3669428592ed5674a72b7ab3" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:13:14.896531 30004 raft_consensus.cc:3037] T 76c9c41a3669428592ed5674a72b7ab3 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:14.899292 435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76c9c41a3669428592ed5674a72b7ab3" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:13:14.899411 30004 raft_consensus.cc:2454] T 76c9c41a3669428592ed5674a72b7ab3 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:13:14.899833 32709 leader_election.cc:302] T 76c9c41a3669428592ed5674a72b7ab3 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 8a666e798fe8477d8dd1e51b385d5ead; no voters:
W20240503 02:13:14.900209 32709 leader_election.cc:341] T 76c9c41a3669428592ed5674a72b7ab3 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:14.900570 2787 raft_consensus.cc:2781] T 76c9c41a3669428592ed5674a72b7ab3 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:14.900758 2787 raft_consensus.cc:686] T 76c9c41a3669428592ed5674a72b7ab3 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Becoming Leader. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Running, Role: LEADER
I20240503 02:13:14.901116 2787 consensus_queue.cc:237] T 76c9c41a3669428592ed5674a72b7ab3 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:14.904584 3159 tablet_bootstrap.cc:654] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:14.905544 547 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:14.909664 3161 tablet_bootstrap.cc:492] T 76c9c41a3669428592ed5674a72b7ab3 P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:14.915993 3166 tablet_bootstrap.cc:492] T 4563386631774e74a4b12fc71714c1a0 P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:14.916220 3166 ts_tablet_manager.cc:1367] T 4563386631774e74a4b12fc71714c1a0 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.044s user 0.000s sys 0.006s
I20240503 02:13:14.917155 3166 raft_consensus.cc:348] T 4563386631774e74a4b12fc71714c1a0 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:14.917431 3166 raft_consensus.cc:374] T 4563386631774e74a4b12fc71714c1a0 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:14.917585 3166 raft_consensus.cc:729] T 4563386631774e74a4b12fc71714c1a0 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:14.917939 3166 consensus_queue.cc:260] T 4563386631774e74a4b12fc71714c1a0 P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:14.918722 3161 ts_tablet_manager.cc:1367] T 76c9c41a3669428592ed5674a72b7ab3 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.053s user 0.002s sys 0.003s
I20240503 02:13:14.920244 3166 ts_tablet_manager.cc:1398] T 4563386631774e74a4b12fc71714c1a0 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:13:14.921780 3167 tablet_bootstrap.cc:492] T ec3891bfc4014b63afef6c259de00ca7 P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:14.922003 3167 ts_tablet_manager.cc:1367] T ec3891bfc4014b63afef6c259de00ca7 P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.003s
I20240503 02:13:14.922781 3167 raft_consensus.cc:348] T ec3891bfc4014b63afef6c259de00ca7 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:14.923038 3167 raft_consensus.cc:374] T ec3891bfc4014b63afef6c259de00ca7 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:14.923154 3167 raft_consensus.cc:729] T ec3891bfc4014b63afef6c259de00ca7 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:14.923417 3167 consensus_queue.cc:260] T ec3891bfc4014b63afef6c259de00ca7 P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:14.924568 3161 raft_consensus.cc:348] T 76c9c41a3669428592ed5674a72b7ab3 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:14.925318 3161 raft_consensus.cc:374] T 76c9c41a3669428592ed5674a72b7ab3 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:14.925588 3161 raft_consensus.cc:729] T 76c9c41a3669428592ed5674a72b7ab3 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:14.926067 3161 consensus_queue.cc:260] T 76c9c41a3669428592ed5674a72b7ab3 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:14.926755 3167 ts_tablet_manager.cc:1398] T ec3891bfc4014b63afef6c259de00ca7 P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240503 02:13:14.930352 3161 ts_tablet_manager.cc:1398] T 76c9c41a3669428592ed5674a72b7ab3 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.011s user 0.000s sys 0.001s
I20240503 02:13:14.933421 3161 tablet_bootstrap.cc:492] T ec3891bfc4014b63afef6c259de00ca7 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:14.942865 3167 tablet_bootstrap.cc:492] T 3f3ec15b1ec043d1a8f252f42a584f63 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:14.944797 3168 tablet_bootstrap.cc:492] T 3f3ec15b1ec043d1a8f252f42a584f63 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:13:14.945180 3168 ts_tablet_manager.cc:1367] T 3f3ec15b1ec043d1a8f252f42a584f63 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.073s user 0.006s sys 0.000s
I20240503 02:13:14.946096 3168 raft_consensus.cc:348] T 3f3ec15b1ec043d1a8f252f42a584f63 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:14.946372 3168 raft_consensus.cc:374] T 3f3ec15b1ec043d1a8f252f42a584f63 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:14.946467 3168 raft_consensus.cc:729] T 3f3ec15b1ec043d1a8f252f42a584f63 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:13:14.946736 3168 consensus_queue.cc:260] T 3f3ec15b1ec043d1a8f252f42a584f63 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:14.947433 3162 tablet_bootstrap.cc:492] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:14.947432 3168 ts_tablet_manager.cc:1398] T 3f3ec15b1ec043d1a8f252f42a584f63 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:14.947852 3161 tablet_bootstrap.cc:654] T ec3891bfc4014b63afef6c259de00ca7 P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:14.948961 3162 tablet_bootstrap.cc:654] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:14.957223 29728 catalog_manager.cc:5484] T 76c9c41a3669428592ed5674a72b7ab3 P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6). New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:14.957553 3052 raft_consensus.cc:482] T 76c9c41a3669428592ed5674a72b7ab3 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:14.957711 3052 raft_consensus.cc:504] T 76c9c41a3669428592ed5674a72b7ab3 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:14.958732 3052 leader_election.cc:288] T 76c9c41a3669428592ed5674a72b7ab3 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:14.959297 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76c9c41a3669428592ed5674a72b7ab3" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:13:14.959568 30004 raft_consensus.cc:2379] T 76c9c41a3669428592ed5674a72b7ab3 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fbbc2be6382e47908ecb7c68c367bd26 in current term 1: Already voted for candidate 8a666e798fe8477d8dd1e51b385d5ead in this term.
I20240503 02:13:14.960922 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76c9c41a3669428592ed5674a72b7ab3" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:13:14.961941 371 leader_election.cc:302] T 76c9c41a3669428592ed5674a72b7ab3 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 155735c7f8b14997ac4c770aee2a9401, 8a666e798fe8477d8dd1e51b385d5ead
I20240503 02:13:14.963107 3052 raft_consensus.cc:3037] T 76c9c41a3669428592ed5674a72b7ab3 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:14.970782 3167 tablet_bootstrap.cc:654] T 3f3ec15b1ec043d1a8f252f42a584f63 P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:14.972829 3162 tablet_bootstrap.cc:492] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:14.973357 3162 ts_tablet_manager.cc:1367] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.005s
I20240503 02:13:14.975675 3052 raft_consensus.cc:2726] T 76c9c41a3669428592ed5674a72b7ab3 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:14.975978 3162 raft_consensus.cc:348] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:14.976287 3162 raft_consensus.cc:374] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:14.976446 3162 raft_consensus.cc:729] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:14.977514 3159 tablet_bootstrap.cc:492] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:13:14.977793 3162 consensus_queue.cc:260] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:14.979166 3162 ts_tablet_manager.cc:1398] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:13:14.979360 3159 ts_tablet_manager.cc:1367] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.095s user 0.009s sys 0.012s
I20240503 02:13:14.980974 3159 raft_consensus.cc:348] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:14.981503 3159 raft_consensus.cc:374] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:14.981614 3159 raft_consensus.cc:729] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:13:14.981869 3159 consensus_queue.cc:260] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:14.985455 3159 ts_tablet_manager.cc:1398] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:13:14.989809 3161 tablet_bootstrap.cc:492] T ec3891bfc4014b63afef6c259de00ca7 P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:14.990408 3161 ts_tablet_manager.cc:1367] T ec3891bfc4014b63afef6c259de00ca7 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.001s
I20240503 02:13:14.991096 3161 raft_consensus.cc:348] T ec3891bfc4014b63afef6c259de00ca7 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:14.991286 3161 raft_consensus.cc:374] T ec3891bfc4014b63afef6c259de00ca7 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:14.991365 3161 raft_consensus.cc:729] T ec3891bfc4014b63afef6c259de00ca7 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:14.996834 3167 tablet_bootstrap.cc:492] T 3f3ec15b1ec043d1a8f252f42a584f63 P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:14.997152 3167 ts_tablet_manager.cc:1367] T 3f3ec15b1ec043d1a8f252f42a584f63 P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.054s user 0.001s sys 0.003s
I20240503 02:13:14.997988 3167 raft_consensus.cc:348] T 3f3ec15b1ec043d1a8f252f42a584f63 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:14.998642 3167 raft_consensus.cc:374] T 3f3ec15b1ec043d1a8f252f42a584f63 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:14.998617 3161 consensus_queue.cc:260] T ec3891bfc4014b63afef6c259de00ca7 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:14.999333 3167 raft_consensus.cc:729] T 3f3ec15b1ec043d1a8f252f42a584f63 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:14.999650 3167 consensus_queue.cc:260] T 3f3ec15b1ec043d1a8f252f42a584f63 P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:15.000285 3167 ts_tablet_manager.cc:1398] T 3f3ec15b1ec043d1a8f252f42a584f63 P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:15.000787 3167 tablet_bootstrap.cc:492] T 4563386631774e74a4b12fc71714c1a0 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:15.004415 3167 tablet_bootstrap.cc:654] T 4563386631774e74a4b12fc71714c1a0 P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:15.008280 3161 ts_tablet_manager.cc:1398] T ec3891bfc4014b63afef6c259de00ca7 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20240503 02:13:15.034883 3132 raft_consensus.cc:482] T 3f3ec15b1ec043d1a8f252f42a584f63 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:15.034968 3167 tablet_bootstrap.cc:492] T 4563386631774e74a4b12fc71714c1a0 P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:15.035135 3167 ts_tablet_manager.cc:1367] T 4563386631774e74a4b12fc71714c1a0 P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.034s user 0.006s sys 0.016s
I20240503 02:13:15.035068 3132 raft_consensus.cc:504] T 3f3ec15b1ec043d1a8f252f42a584f63 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:15.035818 3132 leader_election.cc:288] T 3f3ec15b1ec043d1a8f252f42a584f63 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:13:15.036584 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3ec15b1ec043d1a8f252f42a584f63" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:13:15.036710 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3ec15b1ec043d1a8f252f42a584f63" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:13:15.036867 305 raft_consensus.cc:2454] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 0.
I20240503 02:13:15.036916 30266 raft_consensus.cc:2454] T 3f3ec15b1ec043d1a8f252f42a584f63 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 0.
I20240503 02:13:15.037462 32574 leader_election.cc:302] T 3f3ec15b1ec043d1a8f252f42a584f63 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, 9a4d92bc25374fb3960fde7d10be309d; no voters:
I20240503 02:13:15.038035 3132 raft_consensus.cc:2781] T 3f3ec15b1ec043d1a8f252f42a584f63 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:15.038170 3132 raft_consensus.cc:482] T 3f3ec15b1ec043d1a8f252f42a584f63 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:15.038280 3132 raft_consensus.cc:3037] T 3f3ec15b1ec043d1a8f252f42a584f63 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:15.038563 3167 raft_consensus.cc:348] T 4563386631774e74a4b12fc71714c1a0 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:15.038866 3167 raft_consensus.cc:374] T 4563386631774e74a4b12fc71714c1a0 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:15.039042 3167 raft_consensus.cc:729] T 4563386631774e74a4b12fc71714c1a0 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:15.039400 3167 consensus_queue.cc:260] T 4563386631774e74a4b12fc71714c1a0 P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:15.041500 3132 raft_consensus.cc:504] T 3f3ec15b1ec043d1a8f252f42a584f63 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:15.042470 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3ec15b1ec043d1a8f252f42a584f63" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:13:15.042697 305 raft_consensus.cc:3037] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:15.044109 3167 ts_tablet_manager.cc:1398] T 4563386631774e74a4b12fc71714c1a0 P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240503 02:13:15.046263 305 raft_consensus.cc:2454] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 1.
I20240503 02:13:15.047051 32574 leader_election.cc:302] T 3f3ec15b1ec043d1a8f252f42a584f63 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, 9a4d92bc25374fb3960fde7d10be309d; no voters:
I20240503 02:13:15.047444 3129 raft_consensus.cc:2781] T 3f3ec15b1ec043d1a8f252f42a584f63 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:15.048074 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3ec15b1ec043d1a8f252f42a584f63" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:13:15.048259 30266 raft_consensus.cc:3037] T 3f3ec15b1ec043d1a8f252f42a584f63 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:15.048249 3132 leader_election.cc:288] T 3f3ec15b1ec043d1a8f252f42a584f63 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 election: Requested vote from peers 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:13:15.055166 2946 raft_consensus.cc:482] T 4563386631774e74a4b12fc71714c1a0 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:15.055358 2946 raft_consensus.cc:504] T 4563386631774e74a4b12fc71714c1a0 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:15.056146 2946 leader_election.cc:288] T 4563386631774e74a4b12fc71714c1a0 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:13:15.056658 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4563386631774e74a4b12fc71714c1a0" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:13:15.056697 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4563386631774e74a4b12fc71714c1a0" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:13:15.056896 30005 raft_consensus.cc:2454] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 0.
I20240503 02:13:15.056900 32640 raft_consensus.cc:2454] T 4563386631774e74a4b12fc71714c1a0 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 0.
I20240503 02:13:15.055236 30266 raft_consensus.cc:2454] T 3f3ec15b1ec043d1a8f252f42a584f63 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 1.
I20240503 02:13:15.057986 502 leader_election.cc:302] T 4563386631774e74a4b12fc71714c1a0 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:15.058557 2946 raft_consensus.cc:2781] T 4563386631774e74a4b12fc71714c1a0 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:15.058674 2946 raft_consensus.cc:482] T 4563386631774e74a4b12fc71714c1a0 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:15.058748 2946 raft_consensus.cc:3037] T 4563386631774e74a4b12fc71714c1a0 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:15.063632 2946 raft_consensus.cc:504] T 4563386631774e74a4b12fc71714c1a0 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:15.067695 2946 leader_election.cc:288] T 4563386631774e74a4b12fc71714c1a0 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 election: Requested vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:13:15.067889 3129 raft_consensus.cc:686] T 3f3ec15b1ec043d1a8f252f42a584f63 P 9a4d92bc25374fb3960fde7d10be309d [term 1 LEADER]: Becoming Leader. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Running, Role: LEADER
I20240503 02:13:15.068277 3129 consensus_queue.cc:237] T 3f3ec15b1ec043d1a8f252f42a584f63 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:15.068351 30006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4563386631774e74a4b12fc71714c1a0" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:13:15.068529 30006 raft_consensus.cc:3037] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:15.074172 30006 raft_consensus.cc:2454] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 1.
I20240503 02:13:15.074498 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4563386631774e74a4b12fc71714c1a0" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d"
I20240503 02:13:15.074781 32640 raft_consensus.cc:3037] T 4563386631774e74a4b12fc71714c1a0 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:15.078375 29729 catalog_manager.cc:5484] T 3f3ec15b1ec043d1a8f252f42a584f63 P 9a4d92bc25374fb3960fde7d10be309d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5). New cstate: current_term: 1 leader_uuid: "9a4d92bc25374fb3960fde7d10be309d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:15.079341 502 leader_election.cc:302] T 4563386631774e74a4b12fc71714c1a0 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:15.079788 2946 raft_consensus.cc:2781] T 4563386631774e74a4b12fc71714c1a0 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:15.080770 32640 raft_consensus.cc:2454] T 4563386631774e74a4b12fc71714c1a0 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 1.
I20240503 02:13:15.081475 29734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:13:15.083173 2946 raft_consensus.cc:686] T 4563386631774e74a4b12fc71714c1a0 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 LEADER]: Becoming Leader. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Running, Role: LEADER
I20240503 02:13:15.083499 2946 consensus_queue.cc:237] T 4563386631774e74a4b12fc71714c1a0 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:15.088948 435 raft_consensus.cc:1261] T 76c9c41a3669428592ed5674a72b7ab3 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:15.089665 3076 consensus_queue.cc:1035] T 76c9c41a3669428592ed5674a72b7ab3 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:15.093267 30006 raft_consensus.cc:1261] T 76c9c41a3669428592ed5674a72b7ab3 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:15.093873 2787 consensus_queue.cc:1035] T 76c9c41a3669428592ed5674a72b7ab3 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:15.118454 29729 catalog_manager.cc:5484] T 4563386631774e74a4b12fc71714c1a0 P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: term changed from 0 to 1, leader changed from <none> to f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8). New cstate: current_term: 1 leader_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:15.138773 3052 raft_consensus.cc:482] T ec3891bfc4014b63afef6c259de00ca7 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:15.138965 3052 raft_consensus.cc:504] T ec3891bfc4014b63afef6c259de00ca7 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:15.139812 3052 leader_election.cc:288] T ec3891bfc4014b63afef6c259de00ca7 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:15.141640 29729 catalog_manager.cc:5576] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241) does not have the latest schema for tablet 76c9c41a3669428592ed5674a72b7ab3 (table rebalancer_dynamic_table [id=54927021e06c431686f6a9e7200dfd2d]). Expected version 1 got 0
I20240503 02:13:15.143711 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec3891bfc4014b63afef6c259de00ca7" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:13:15.144023 32640 raft_consensus.cc:2454] T ec3891bfc4014b63afef6c259de00ca7 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 0.
I20240503 02:13:15.144433 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec3891bfc4014b63afef6c259de00ca7" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:13:15.144716 305 raft_consensus.cc:2454] T ec3891bfc4014b63afef6c259de00ca7 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 0.
I20240503 02:13:15.146018 371 leader_election.cc:302] T ec3891bfc4014b63afef6c259de00ca7 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:15.146346 3052 raft_consensus.cc:2781] T ec3891bfc4014b63afef6c259de00ca7 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:15.146435 3052 raft_consensus.cc:482] T ec3891bfc4014b63afef6c259de00ca7 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:15.146553 3052 raft_consensus.cc:3037] T ec3891bfc4014b63afef6c259de00ca7 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:15.149299 3052 raft_consensus.cc:504] T ec3891bfc4014b63afef6c259de00ca7 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:15.151461 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec3891bfc4014b63afef6c259de00ca7" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d"
I20240503 02:13:15.151698 32640 raft_consensus.cc:3037] T ec3891bfc4014b63afef6c259de00ca7 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:15.154654 32640 raft_consensus.cc:2454] T ec3891bfc4014b63afef6c259de00ca7 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 1.
I20240503 02:13:15.155531 3052 leader_election.cc:288] T ec3891bfc4014b63afef6c259de00ca7 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Requested vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:15.155951 371 leader_election.cc:302] T ec3891bfc4014b63afef6c259de00ca7 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:15.156680 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec3891bfc4014b63afef6c259de00ca7" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:13:15.156910 305 raft_consensus.cc:3037] T ec3891bfc4014b63afef6c259de00ca7 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:15.160296 305 raft_consensus.cc:2454] T ec3891bfc4014b63afef6c259de00ca7 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 1.
I20240503 02:13:15.161724 3052 raft_consensus.cc:2781] T ec3891bfc4014b63afef6c259de00ca7 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:15.161947 3052 raft_consensus.cc:686] T ec3891bfc4014b63afef6c259de00ca7 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Becoming Leader. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Running, Role: LEADER
I20240503 02:13:15.162240 3052 consensus_queue.cc:237] T ec3891bfc4014b63afef6c259de00ca7 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:15.162488 29729 catalog_manager.cc:5576] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873) does not have the latest schema for tablet 76c9c41a3669428592ed5674a72b7ab3 (table rebalancer_dynamic_table [id=54927021e06c431686f6a9e7200dfd2d]). Expected version 1 got 0
I20240503 02:13:15.171916 3192 tablet.cc:1722] T 76c9c41a3669428592ed5674a72b7ab3 P 155735c7f8b14997ac4c770aee2a9401: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:15.175089 3192 tablet.cc:1620] T 76c9c41a3669428592ed5674a72b7ab3 P 155735c7f8b14997ac4c770aee2a9401: MemRowSet was empty: no flush needed.
I20240503 02:13:15.177002 29729 catalog_manager.cc:5576] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873) does not have the latest schema for tablet 76c9c41a3669428592ed5674a72b7ab3 (table rebalancer_dynamic_table [id=54927021e06c431686f6a9e7200dfd2d]). Expected version 1 got 0
I20240503 02:13:15.190136 29729 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=54927021e06c431686f6a9e7200dfd2d] alter complete (version 1)
I20240503 02:13:15.199317 3190 tablet.cc:1722] T 76c9c41a3669428592ed5674a72b7ab3 P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:15.200155 3191 tablet.cc:1722] T 76c9c41a3669428592ed5674a72b7ab3 P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:15.201414 3191 tablet.cc:1620] T 76c9c41a3669428592ed5674a72b7ab3 P fbbc2be6382e47908ecb7c68c367bd26: MemRowSet was empty: no flush needed.
I20240503 02:13:15.201869 3190 tablet.cc:1620] T 76c9c41a3669428592ed5674a72b7ab3 P 8a666e798fe8477d8dd1e51b385d5ead: MemRowSet was empty: no flush needed.
I20240503 02:13:15.218914 29729 catalog_manager.cc:5576] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241) does not have the latest schema for tablet 76c9c41a3669428592ed5674a72b7ab3 (table rebalancer_dynamic_table [id=54927021e06c431686f6a9e7200dfd2d]). Expected version 1 got 0
I20240503 02:13:15.225574 29734 catalog_manager.cc:5484] T ec3891bfc4014b63afef6c259de00ca7 P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7). New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:15.299253 29729 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:13:15.309244 3190 tablet.cc:1722] T 76c9c41a3669428592ed5674a72b7ab3 P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:15.309609 3192 tablet.cc:1722] T 76c9c41a3669428592ed5674a72b7ab3 P 155735c7f8b14997ac4c770aee2a9401: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:15.309854 3190 tablet.cc:1620] T 76c9c41a3669428592ed5674a72b7ab3 P 8a666e798fe8477d8dd1e51b385d5ead: MemRowSet was empty: no flush needed.
I20240503 02:13:15.310091 3192 tablet.cc:1620] T 76c9c41a3669428592ed5674a72b7ab3 P 155735c7f8b14997ac4c770aee2a9401: MemRowSet was empty: no flush needed.
I20240503 02:13:15.312414 3191 tablet.cc:1722] T 76c9c41a3669428592ed5674a72b7ab3 P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:15.312979 3191 tablet.cc:1620] T 76c9c41a3669428592ed5674a72b7ab3 P fbbc2be6382e47908ecb7c68c367bd26: MemRowSet was empty: no flush needed.
I20240503 02:13:15.331454 29729 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=54927021e06c431686f6a9e7200dfd2d] alter complete (version 2)
I20240503 02:13:15.437111 29729 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:13:15.446398 3190 tablet.cc:1722] T 76c9c41a3669428592ed5674a72b7ab3 P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:15.447775 3191 tablet.cc:1722] T 76c9c41a3669428592ed5674a72b7ab3 P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:15.450829 3192 tablet.cc:1722] T 76c9c41a3669428592ed5674a72b7ab3 P 155735c7f8b14997ac4c770aee2a9401: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:15.464620 29734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table_new_name [id=54927021e06c431686f6a9e7200dfd2d] alter complete (version 3)
I20240503 02:13:15.489411 3132 consensus_queue.cc:1035] T 3f3ec15b1ec043d1a8f252f42a584f63 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:15.507764 3132 consensus_queue.cc:1035] T 3f3ec15b1ec043d1a8f252f42a584f63 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:15.523488 2946 consensus_queue.cc:1035] T 4563386631774e74a4b12fc71714c1a0 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:15.560237 3068 consensus_queue.cc:1035] T 4563386631774e74a4b12fc71714c1a0 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:15.570371 29734 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:15.570590 29734 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:15.575182 29734 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet 76c9c41a3669428592ed5674a72b7ab3
I20240503 02:13:15.576201 29984 tablet_service.cc:1514] Processing DeleteTablet for tablet 76c9c41a3669428592ed5674a72b7ab3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:15 UTC) from {username='slave'} at 127.0.0.1:52158
I20240503 02:13:15.576507 32754 tablet_service.cc:1514] Processing DeleteTablet for tablet 76c9c41a3669428592ed5674a72b7ab3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:15 UTC) from {username='slave'} at 127.0.0.1:46360
I20240503 02:13:15.577001 416 tablet_service.cc:1514] Processing DeleteTablet for tablet 76c9c41a3669428592ed5674a72b7ab3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:15 UTC) from {username='slave'} at 127.0.0.1:50814
I20240503 02:13:15.581826 3238 tablet_replica.cc:330] T 76c9c41a3669428592ed5674a72b7ab3 P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:13:15.582032 3238 raft_consensus.cc:2229] T 76c9c41a3669428592ed5674a72b7ab3 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:15.582208 3238 raft_consensus.cc:2258] T 76c9c41a3669428592ed5674a72b7ab3 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:15.582686 3240 tablet_replica.cc:330] T 76c9c41a3669428592ed5674a72b7ab3 P fbbc2be6382e47908ecb7c68c367bd26: stopping tablet replica
I20240503 02:13:15.582917 3240 raft_consensus.cc:2229] T 76c9c41a3669428592ed5674a72b7ab3 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:15.583127 3240 raft_consensus.cc:2258] T 76c9c41a3669428592ed5674a72b7ab3 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:15.583623 3239 tablet_replica.cc:330] T 76c9c41a3669428592ed5674a72b7ab3 P 8a666e798fe8477d8dd1e51b385d5ead: stopping tablet replica
I20240503 02:13:15.583710 3238 ts_tablet_manager.cc:1875] T 76c9c41a3669428592ed5674a72b7ab3 P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:15.583904 3239 raft_consensus.cc:2229] T 76c9c41a3669428592ed5674a72b7ab3 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:15.584218 3240 ts_tablet_manager.cc:1875] T 76c9c41a3669428592ed5674a72b7ab3 P fbbc2be6382e47908ecb7c68c367bd26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:15.584535 3239 raft_consensus.cc:2258] T 76c9c41a3669428592ed5674a72b7ab3 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:15.585527 3239 ts_tablet_manager.cc:1875] T 76c9c41a3669428592ed5674a72b7ab3 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:15.590939 3180 consensus_queue.cc:1035] T ec3891bfc4014b63afef6c259de00ca7 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:15.600240 3239 ts_tablet_manager.cc:1888] T 76c9c41a3669428592ed5674a72b7ab3 P 8a666e798fe8477d8dd1e51b385d5ead: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:15.600446 3239 log.cc:1196] T 76c9c41a3669428592ed5674a72b7ab3 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal/wals/76c9c41a3669428592ed5674a72b7ab3
I20240503 02:13:15.601095 3239 ts_tablet_manager.cc:1909] T 76c9c41a3669428592ed5674a72b7ab3 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting consensus metadata
I20240503 02:13:15.603866 3238 ts_tablet_manager.cc:1888] T 76c9c41a3669428592ed5674a72b7ab3 P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:15.603924 29719 catalog_manager.cc:4830] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): tablet 76c9c41a3669428592ed5674a72b7ab3 (table rebalancer_dynamic_table_new_name [id=54927021e06c431686f6a9e7200dfd2d]) successfully deleted
I20240503 02:13:15.604027 3238 log.cc:1196] T 76c9c41a3669428592ed5674a72b7ab3 P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/76c9c41a3669428592ed5674a72b7ab3
I20240503 02:13:15.604461 3238 ts_tablet_manager.cc:1909] T 76c9c41a3669428592ed5674a72b7ab3 P 155735c7f8b14997ac4c770aee2a9401: Deleting consensus metadata
I20240503 02:13:15.606736 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet 76c9c41a3669428592ed5674a72b7ab3 (table rebalancer_dynamic_table_new_name [id=54927021e06c431686f6a9e7200dfd2d]) successfully deleted
I20240503 02:13:15.609483 3052 consensus_queue.cc:1035] T ec3891bfc4014b63afef6c259de00ca7 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:15.628238 3240 ts_tablet_manager.cc:1888] T 76c9c41a3669428592ed5674a72b7ab3 P fbbc2be6382e47908ecb7c68c367bd26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:15.628372 3240 log.cc:1196] T 76c9c41a3669428592ed5674a72b7ab3 P fbbc2be6382e47908ecb7c68c367bd26: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal/wals/76c9c41a3669428592ed5674a72b7ab3
I20240503 02:13:15.628839 3240 ts_tablet_manager.cc:1909] T 76c9c41a3669428592ed5674a72b7ab3 P fbbc2be6382e47908ecb7c68c367bd26: Deleting consensus metadata
I20240503 02:13:15.629741 29719 catalog_manager.cc:4830] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): tablet 76c9c41a3669428592ed5674a72b7ab3 (table rebalancer_dynamic_table_new_name [id=54927021e06c431686f6a9e7200dfd2d]) successfully deleted
I20240503 02:13:15.681150 29730 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:13:15.699065 416 tablet_service.cc:1467] Processing CreateTablet for tablet d66a33b0d22142b19b502dfedd6c4cdd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=680497bb39cf42e9aa4264c5ace9ca85]), partition=
I20240503 02:13:15.699342 547 tablet_service.cc:1467] Processing CreateTablet for tablet d66a33b0d22142b19b502dfedd6c4cdd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=680497bb39cf42e9aa4264c5ace9ca85]), partition=
I20240503 02:13:15.699512 416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d66a33b0d22142b19b502dfedd6c4cdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:15.699662 547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d66a33b0d22142b19b502dfedd6c4cdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:15.701789 30116 tablet_service.cc:1467] Processing CreateTablet for tablet d66a33b0d22142b19b502dfedd6c4cdd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=680497bb39cf42e9aa4264c5ace9ca85]), partition=
I20240503 02:13:15.702108 30116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d66a33b0d22142b19b502dfedd6c4cdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:15.716379 29984 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:13:15.734440 30245 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:15.753329 29853 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:13:15.766947 29730 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_extra_table_38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:15.784482 548 tablet_service.cc:1467] Processing CreateTablet for tablet b8b119c0d8044f2ab19b2100950270d6 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=e4b9d13872ad4d27841df71e38d5482f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:15.784705 546 tablet_service.cc:1467] Processing CreateTablet for tablet c87e7cbed0e843f5b5bb9bddef598cdb (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=e4b9d13872ad4d27841df71e38d5482f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:15.785538 548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8b119c0d8044f2ab19b2100950270d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:15.786592 30245 tablet_service.cc:1467] Processing CreateTablet for tablet 5a44b34b585b47528c22fc678289b01a (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=e4b9d13872ad4d27841df71e38d5482f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:15.787087 30245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a44b34b585b47528c22fc678289b01a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:15.790571 32754 tablet_service.cc:1467] Processing CreateTablet for tablet c87e7cbed0e843f5b5bb9bddef598cdb (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=e4b9d13872ad4d27841df71e38d5482f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:15.790993 32754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c87e7cbed0e843f5b5bb9bddef598cdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:15.793372 32755 tablet_service.cc:1467] Processing CreateTablet for tablet 5a44b34b585b47528c22fc678289b01a (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=e4b9d13872ad4d27841df71e38d5482f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:15.793725 32755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a44b34b585b47528c22fc678289b01a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:15.784711 545 tablet_service.cc:1467] Processing CreateTablet for tablet 5a44b34b585b47528c22fc678289b01a (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=e4b9d13872ad4d27841df71e38d5482f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:15.799342 32752 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:13:15.799314 32619 tablet_service.cc:1467] Processing CreateTablet for tablet b8b119c0d8044f2ab19b2100950270d6 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=e4b9d13872ad4d27841df71e38d5482f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:15.799849 32619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8b119c0d8044f2ab19b2100950270d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:15.801635 546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c87e7cbed0e843f5b5bb9bddef598cdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:15.811829 545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a44b34b585b47528c22fc678289b01a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:15.816761 3272 tablet_bootstrap.cc:492] T d66a33b0d22142b19b502dfedd6c4cdd P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:15.828748 3272 tablet_bootstrap.cc:654] T d66a33b0d22142b19b502dfedd6c4cdd P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:15.833715 3267 tablet_bootstrap.cc:492] T d66a33b0d22142b19b502dfedd6c4cdd P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:13:15.835283 3267 tablet_bootstrap.cc:654] T d66a33b0d22142b19b502dfedd6c4cdd P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:15.843429 3274 tablet_bootstrap.cc:492] T c87e7cbed0e843f5b5bb9bddef598cdb P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
W20240503 02:13:15.837724 30072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.1:36719, user_credentials={real_user=slave}} blocked reactor thread for 130110us
I20240503 02:13:15.846019 3274 tablet_bootstrap.cc:654] T c87e7cbed0e843f5b5bb9bddef598cdb P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:15.848778 30114 tablet_service.cc:1467] Processing CreateTablet for tablet c87e7cbed0e843f5b5bb9bddef598cdb (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=e4b9d13872ad4d27841df71e38d5482f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:15.849226 30114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c87e7cbed0e843f5b5bb9bddef598cdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:15.849496 30116 tablet_service.cc:1467] Processing CreateTablet for tablet b8b119c0d8044f2ab19b2100950270d6 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=e4b9d13872ad4d27841df71e38d5482f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:15.849892 30116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8b119c0d8044f2ab19b2100950270d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:15.852906 3275 tablet_bootstrap.cc:492] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:13:15.854795 32621 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:15.856870 3275 tablet_bootstrap.cc:654] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:15.858323 3276 tablet_bootstrap.cc:492] T b8b119c0d8044f2ab19b2100950270d6 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:15.860208 3276 tablet_bootstrap.cc:654] T b8b119c0d8044f2ab19b2100950270d6 P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:15.867659 3274 tablet_bootstrap.cc:492] T c87e7cbed0e843f5b5bb9bddef598cdb P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:15.867830 3274 ts_tablet_manager.cc:1367] T c87e7cbed0e843f5b5bb9bddef598cdb P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
I20240503 02:13:15.868613 3274 raft_consensus.cc:348] T c87e7cbed0e843f5b5bb9bddef598cdb P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:15.868835 3274 raft_consensus.cc:374] T c87e7cbed0e843f5b5bb9bddef598cdb P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:15.868924 3274 raft_consensus.cc:729] T c87e7cbed0e843f5b5bb9bddef598cdb P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:15.869160 3274 consensus_queue.cc:260] T c87e7cbed0e843f5b5bb9bddef598cdb P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:15.872092 3274 ts_tablet_manager.cc:1398] T c87e7cbed0e843f5b5bb9bddef598cdb P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:13:15.875047 3274 tablet_bootstrap.cc:492] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:15.890774 3274 tablet_bootstrap.cc:654] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:15.904047 3274 tablet_bootstrap.cc:492] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:15.904778 3274 ts_tablet_manager.cc:1367] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.003s
I20240503 02:13:15.905803 3276 tablet_bootstrap.cc:492] T b8b119c0d8044f2ab19b2100950270d6 P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:15.905978 3276 ts_tablet_manager.cc:1367] T b8b119c0d8044f2ab19b2100950270d6 P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.048s user 0.002s sys 0.004s
I20240503 02:13:15.907045 3276 raft_consensus.cc:348] T b8b119c0d8044f2ab19b2100950270d6 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:15.907251 3276 raft_consensus.cc:374] T b8b119c0d8044f2ab19b2100950270d6 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:15.907341 3276 raft_consensus.cc:729] T b8b119c0d8044f2ab19b2100950270d6 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:15.907356 3274 raft_consensus.cc:348] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:15.907960 3274 raft_consensus.cc:374] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:15.908296 3274 raft_consensus.cc:729] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:15.907625 3276 consensus_queue.cc:260] T b8b119c0d8044f2ab19b2100950270d6 P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:15.910928 3274 consensus_queue.cc:260] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:15.911613 3276 ts_tablet_manager.cc:1398] T b8b119c0d8044f2ab19b2100950270d6 P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20240503 02:13:15.917536 3265 tablet_bootstrap.cc:492] T d66a33b0d22142b19b502dfedd6c4cdd P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:15.918823 3274 ts_tablet_manager.cc:1398] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20240503 02:13:15.921245 3267 tablet_bootstrap.cc:492] T d66a33b0d22142b19b502dfedd6c4cdd P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:13:15.921443 3267 ts_tablet_manager.cc:1367] T d66a33b0d22142b19b502dfedd6c4cdd P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.088s user 0.002s sys 0.002s
I20240503 02:13:15.925736 3267 raft_consensus.cc:348] T d66a33b0d22142b19b502dfedd6c4cdd P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:15.945111 3272 tablet_bootstrap.cc:492] T d66a33b0d22142b19b502dfedd6c4cdd P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:15.945384 3272 ts_tablet_manager.cc:1367] T d66a33b0d22142b19b502dfedd6c4cdd P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.129s user 0.005s sys 0.000s
I20240503 02:13:15.946319 3272 raft_consensus.cc:348] T d66a33b0d22142b19b502dfedd6c4cdd P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:15.946656 3272 raft_consensus.cc:374] T d66a33b0d22142b19b502dfedd6c4cdd P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:15.946808 3272 raft_consensus.cc:729] T d66a33b0d22142b19b502dfedd6c4cdd P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:15.947149 3272 consensus_queue.cc:260] T d66a33b0d22142b19b502dfedd6c4cdd P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:15.953150 3275 tablet_bootstrap.cc:492] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:13:15.953313 3275 ts_tablet_manager.cc:1367] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.100s user 0.010s sys 0.001s
I20240503 02:13:15.953971 3272 ts_tablet_manager.cc:1398] T d66a33b0d22142b19b502dfedd6c4cdd P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:13:15.956844 3275 raft_consensus.cc:348] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:15.958971 3275 raft_consensus.cc:374] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:15.959116 3275 raft_consensus.cc:729] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:13:15.962035 3275 consensus_queue.cc:260] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:15.962750 3275 ts_tablet_manager.cc:1398] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240503 02:13:15.977608 416 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:15.982198 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:15.938952 30116 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:16.005108 3267 raft_consensus.cc:374] T d66a33b0d22142b19b502dfedd6c4cdd P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:16.005200 3267 raft_consensus.cc:729] T d66a33b0d22142b19b502dfedd6c4cdd P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:13:16.005465 3267 consensus_queue.cc:260] T d66a33b0d22142b19b502dfedd6c4cdd P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:16.009289 3267 ts_tablet_manager.cc:1398] T d66a33b0d22142b19b502dfedd6c4cdd P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.084s user 0.001s sys 0.001s
I20240503 02:13:16.009532 3267 tablet_bootstrap.cc:492] T c87e7cbed0e843f5b5bb9bddef598cdb P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:13:16.010931 3267 tablet_bootstrap.cc:654] T c87e7cbed0e843f5b5bb9bddef598cdb P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:16.023612 3180 raft_consensus.cc:482] T d66a33b0d22142b19b502dfedd6c4cdd P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:16.023823 3180 raft_consensus.cc:504] T d66a33b0d22142b19b502dfedd6c4cdd P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:16.025758 3265 tablet_bootstrap.cc:654] T d66a33b0d22142b19b502dfedd6c4cdd P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:16.031105 3267 tablet_bootstrap.cc:492] T c87e7cbed0e843f5b5bb9bddef598cdb P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:13:16.032073 3267 ts_tablet_manager.cc:1367] T c87e7cbed0e843f5b5bb9bddef598cdb P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20240503 02:13:16.033694 3267 raft_consensus.cc:348] T c87e7cbed0e843f5b5bb9bddef598cdb P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:16.033845 3180 leader_election.cc:288] T d66a33b0d22142b19b502dfedd6c4cdd P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:13:16.036969 3267 raft_consensus.cc:374] T c87e7cbed0e843f5b5bb9bddef598cdb P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:16.034598 30135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d66a33b0d22142b19b502dfedd6c4cdd" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:13:16.038177 3267 raft_consensus.cc:729] T c87e7cbed0e843f5b5bb9bddef598cdb P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:13:16.038256 30135 raft_consensus.cc:2454] T d66a33b0d22142b19b502dfedd6c4cdd P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 0.
I20240503 02:13:16.038830 370 leader_election.cc:302] T d66a33b0d22142b19b502dfedd6c4cdd P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:16.039188 3292 raft_consensus.cc:2781] T d66a33b0d22142b19b502dfedd6c4cdd P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:16.039300 3292 raft_consensus.cc:482] T d66a33b0d22142b19b502dfedd6c4cdd P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:16.039229 3267 consensus_queue.cc:260] T c87e7cbed0e843f5b5bb9bddef598cdb P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:16.039425 3292 raft_consensus.cc:3037] T d66a33b0d22142b19b502dfedd6c4cdd P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:16.039791 3267 ts_tablet_manager.cc:1398] T c87e7cbed0e843f5b5bb9bddef598cdb P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 02:13:16.040051 3267 tablet_bootstrap.cc:492] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:13:16.041476 3267 tablet_bootstrap.cc:654] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:16.042554 3292 raft_consensus.cc:504] T d66a33b0d22142b19b502dfedd6c4cdd P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:16.043892 30135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d66a33b0d22142b19b502dfedd6c4cdd" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:13:16.044076 30135 raft_consensus.cc:3037] T d66a33b0d22142b19b502dfedd6c4cdd P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:16.046836 30135 raft_consensus.cc:2454] T d66a33b0d22142b19b502dfedd6c4cdd P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 1.
I20240503 02:13:16.047482 370 leader_election.cc:302] T d66a33b0d22142b19b502dfedd6c4cdd P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:16.047799 3180 raft_consensus.cc:2781] T d66a33b0d22142b19b502dfedd6c4cdd P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:16.047928 3180 raft_consensus.cc:686] T d66a33b0d22142b19b502dfedd6c4cdd P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Becoming Leader. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Running, Role: LEADER
I20240503 02:13:16.048254 3180 consensus_queue.cc:237] T d66a33b0d22142b19b502dfedd6c4cdd P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:16.054414 3292 leader_election.cc:288] T d66a33b0d22142b19b502dfedd6c4cdd P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Requested vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:13:16.067960 29725 catalog_manager.cc:5484] T d66a33b0d22142b19b502dfedd6c4cdd P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7). New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:16.068686 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d66a33b0d22142b19b502dfedd6c4cdd" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:16.069415 566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d66a33b0d22142b19b502dfedd6c4cdd" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
W20240503 02:13:16.070111 370 leader_election.cc:341] T d66a33b0d22142b19b502dfedd6c4cdd P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:13:16.070442 370 leader_election.cc:341] T d66a33b0d22142b19b502dfedd6c4cdd P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:16.098132 3265 tablet_bootstrap.cc:492] T d66a33b0d22142b19b502dfedd6c4cdd P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:16.098313 3265 ts_tablet_manager.cc:1367] T d66a33b0d22142b19b502dfedd6c4cdd P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.181s user 0.003s sys 0.000s
I20240503 02:13:16.099179 3265 raft_consensus.cc:348] T d66a33b0d22142b19b502dfedd6c4cdd P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:16.099411 3265 raft_consensus.cc:374] T d66a33b0d22142b19b502dfedd6c4cdd P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:16.099514 3265 raft_consensus.cc:729] T d66a33b0d22142b19b502dfedd6c4cdd P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:16.099776 3265 consensus_queue.cc:260] T d66a33b0d22142b19b502dfedd6c4cdd P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:16.100474 3265 ts_tablet_manager.cc:1398] T d66a33b0d22142b19b502dfedd6c4cdd P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:13:16.100739 3265 tablet_bootstrap.cc:492] T 5a44b34b585b47528c22fc678289b01a P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:16.102484 3265 tablet_bootstrap.cc:654] T 5a44b34b585b47528c22fc678289b01a P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:16.111637 3267 tablet_bootstrap.cc:492] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:13:16.111788 3267 ts_tablet_manager.cc:1367] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.072s user 0.016s sys 0.013s
I20240503 02:13:16.113454 3267 raft_consensus.cc:348] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:16.113670 3267 raft_consensus.cc:374] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:16.113726 3267 raft_consensus.cc:729] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:13:16.113941 3267 consensus_queue.cc:260] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:16.115530 3267 ts_tablet_manager.cc:1398] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:13:16.123255 3265 tablet_bootstrap.cc:492] T 5a44b34b585b47528c22fc678289b01a P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:16.123450 3265 ts_tablet_manager.cc:1367] T 5a44b34b585b47528c22fc678289b01a P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20240503 02:13:16.124575 3265 raft_consensus.cc:348] T 5a44b34b585b47528c22fc678289b01a P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:16.124820 3265 raft_consensus.cc:374] T 5a44b34b585b47528c22fc678289b01a P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:16.124907 3265 raft_consensus.cc:729] T 5a44b34b585b47528c22fc678289b01a P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:16.127835 3265 consensus_queue.cc:260] T 5a44b34b585b47528c22fc678289b01a P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:16.129774 3265 ts_tablet_manager.cc:1398] T 5a44b34b585b47528c22fc678289b01a P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240503 02:13:16.130136 3265 tablet_bootstrap.cc:492] T b8b119c0d8044f2ab19b2100950270d6 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:16.133770 3265 tablet_bootstrap.cc:654] T b8b119c0d8044f2ab19b2100950270d6 P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:16.146585 3265 tablet_bootstrap.cc:492] T b8b119c0d8044f2ab19b2100950270d6 P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:16.146917 3265 ts_tablet_manager.cc:1367] T b8b119c0d8044f2ab19b2100950270d6 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.004s
I20240503 02:13:16.148274 3265 raft_consensus.cc:348] T b8b119c0d8044f2ab19b2100950270d6 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:16.148588 3265 raft_consensus.cc:374] T b8b119c0d8044f2ab19b2100950270d6 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:16.148746 3265 raft_consensus.cc:729] T b8b119c0d8044f2ab19b2100950270d6 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:16.149083 3265 consensus_queue.cc:260] T b8b119c0d8044f2ab19b2100950270d6 P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:16.151455 3265 ts_tablet_manager.cc:1398] T b8b119c0d8044f2ab19b2100950270d6 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:13:16.152125 3265 tablet_bootstrap.cc:492] T c87e7cbed0e843f5b5bb9bddef598cdb P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:16.157284 3265 tablet_bootstrap.cc:654] T c87e7cbed0e843f5b5bb9bddef598cdb P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:16.164472 3266 raft_consensus.cc:482] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:16.164625 3266 raft_consensus.cc:504] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:16.165421 3266 leader_election.cc:288] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:13:16.166781 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b119c0d8044f2ab19b2100950270d6" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:13:16.167088 32640 raft_consensus.cc:2454] T b8b119c0d8044f2ab19b2100950270d6 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 0.
I20240503 02:13:16.167951 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b119c0d8044f2ab19b2100950270d6" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:16.168268 567 raft_consensus.cc:2454] T b8b119c0d8044f2ab19b2100950270d6 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 0.
I20240503 02:13:16.168749 30069 leader_election.cc:302] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:16.169057 3266 raft_consensus.cc:2781] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:16.169142 3266 raft_consensus.cc:482] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:16.169214 3266 raft_consensus.cc:3037] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:16.171828 3266 raft_consensus.cc:504] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:16.172662 3266 leader_election.cc:288] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Requested vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:13:16.172693 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b119c0d8044f2ab19b2100950270d6" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d"
I20240503 02:13:16.173012 32640 raft_consensus.cc:3037] T b8b119c0d8044f2ab19b2100950270d6 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:16.177260 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b119c0d8044f2ab19b2100950270d6" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:13:16.177522 567 raft_consensus.cc:3037] T b8b119c0d8044f2ab19b2100950270d6 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:16.180958 3265 tablet_bootstrap.cc:492] T c87e7cbed0e843f5b5bb9bddef598cdb P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:16.181211 3265 ts_tablet_manager.cc:1367] T c87e7cbed0e843f5b5bb9bddef598cdb P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.016s
I20240503 02:13:16.182072 567 raft_consensus.cc:2454] T b8b119c0d8044f2ab19b2100950270d6 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 1.
I20240503 02:13:16.182088 3265 raft_consensus.cc:348] T c87e7cbed0e843f5b5bb9bddef598cdb P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:16.182348 3076 raft_consensus.cc:482] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:16.182520 3265 raft_consensus.cc:374] T c87e7cbed0e843f5b5bb9bddef598cdb P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:16.182512 30069 leader_election.cc:302] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:16.182504 3076 raft_consensus.cc:504] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:16.182776 3266 raft_consensus.cc:2781] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:16.182885 3266 raft_consensus.cc:686] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Becoming Leader. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Running, Role: LEADER
I20240503 02:13:16.183164 3076 leader_election.cc:288] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:13:16.183100 3266 consensus_queue.cc:237] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:16.183481 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a44b34b585b47528c22fc678289b01a" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:16.183940 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a44b34b585b47528c22fc678289b01a" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:13:16.184171 30266 raft_consensus.cc:2454] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:13:16.185053 32709 leader_election.cc:302] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, 8a666e798fe8477d8dd1e51b385d5ead; no voters:
I20240503 02:13:16.185375 3076 raft_consensus.cc:2781] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:16.185492 3076 raft_consensus.cc:482] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:16.185570 3076 raft_consensus.cc:3037] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:16.188210 3076 raft_consensus.cc:504] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:16.188474 567 raft_consensus.cc:2454] T 5a44b34b585b47528c22fc678289b01a P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:13:16.189254 566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a44b34b585b47528c22fc678289b01a" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:13:16.189441 566 raft_consensus.cc:3037] T 5a44b34b585b47528c22fc678289b01a P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:16.189442 32640 raft_consensus.cc:2454] T b8b119c0d8044f2ab19b2100950270d6 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 1.
I20240503 02:13:16.189798 3265 raft_consensus.cc:729] T c87e7cbed0e843f5b5bb9bddef598cdb P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:16.190076 3265 consensus_queue.cc:260] T c87e7cbed0e843f5b5bb9bddef598cdb P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:16.190686 3265 ts_tablet_manager.cc:1398] T c87e7cbed0e843f5b5bb9bddef598cdb P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240503 02:13:16.192291 566 raft_consensus.cc:2454] T 5a44b34b585b47528c22fc678289b01a P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:13:16.192699 32708 leader_election.cc:302] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:16.194976 2931 raft_consensus.cc:2781] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:16.196192 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a44b34b585b47528c22fc678289b01a" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:13:16.196413 30266 raft_consensus.cc:3037] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:16.199302 30266 raft_consensus.cc:2454] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:13:16.201426 3076 leader_election.cc:288] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Requested vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:13:16.208582 2931 raft_consensus.cc:686] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Becoming Leader. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Running, Role: LEADER
I20240503 02:13:16.208961 2931 consensus_queue.cc:237] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:16.211449 29729 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:13:16.226661 29725 catalog_manager.cc:5484] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6). New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:16.226853 29729 catalog_manager.cc:5484] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9094240860e43bfb129193f3bfd5be6 (127.22.108.3). New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:16.239574 566 raft_consensus.cc:3037] T d66a33b0d22142b19b502dfedd6c4cdd P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:16.241432 30135 raft_consensus.cc:1261] T d66a33b0d22142b19b502dfedd6c4cdd P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:16.242300 566 raft_consensus.cc:1261] T d66a33b0d22142b19b502dfedd6c4cdd P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:16.243119 3292 consensus_queue.cc:1035] T d66a33b0d22142b19b502dfedd6c4cdd P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:16.243373 3180 consensus_queue.cc:1035] T d66a33b0d22142b19b502dfedd6c4cdd P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:16.248785 29729 catalog_manager.cc:5576] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679) does not have the latest schema for tablet d66a33b0d22142b19b502dfedd6c4cdd (table rebalancer_dynamic_table [id=680497bb39cf42e9aa4264c5ace9ca85]). Expected version 1 got 0
I20240503 02:13:16.254201 3312 tablet.cc:1722] T d66a33b0d22142b19b502dfedd6c4cdd P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:16.255981 3312 tablet.cc:1620] T d66a33b0d22142b19b502dfedd6c4cdd P fbbc2be6382e47908ecb7c68c367bd26: MemRowSet was empty: no flush needed.
I20240503 02:13:16.258586 3313 tablet.cc:1722] T d66a33b0d22142b19b502dfedd6c4cdd P a9094240860e43bfb129193f3bfd5be6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:16.259059 3313 tablet.cc:1620] T d66a33b0d22142b19b502dfedd6c4cdd P a9094240860e43bfb129193f3bfd5be6: MemRowSet was empty: no flush needed.
I20240503 02:13:16.259505 29729 catalog_manager.cc:5576] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679) does not have the latest schema for tablet d66a33b0d22142b19b502dfedd6c4cdd (table rebalancer_dynamic_table [id=680497bb39cf42e9aa4264c5ace9ca85]). Expected version 1 got 0
I20240503 02:13:16.266667 2931 raft_consensus.cc:482] T c87e7cbed0e843f5b5bb9bddef598cdb P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:16.266846 2931 raft_consensus.cc:504] T c87e7cbed0e843f5b5bb9bddef598cdb P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:16.267534 2931 leader_election.cc:288] T c87e7cbed0e843f5b5bb9bddef598cdb P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:13:16.269567 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c87e7cbed0e843f5b5bb9bddef598cdb" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:16.269855 567 raft_consensus.cc:2454] T c87e7cbed0e843f5b5bb9bddef598cdb P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:13:16.270494 32708 leader_election.cc:302] T c87e7cbed0e843f5b5bb9bddef598cdb P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:16.270812 3299 raft_consensus.cc:2781] T c87e7cbed0e843f5b5bb9bddef598cdb P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:16.270741 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c87e7cbed0e843f5b5bb9bddef598cdb" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:13:16.270898 3299 raft_consensus.cc:482] T c87e7cbed0e843f5b5bb9bddef598cdb P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:16.270933 30134 raft_consensus.cc:2454] T c87e7cbed0e843f5b5bb9bddef598cdb P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:13:16.271024 3299 raft_consensus.cc:3037] T c87e7cbed0e843f5b5bb9bddef598cdb P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:16.275149 29725 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=680497bb39cf42e9aa4264c5ace9ca85] alter complete (version 1)
I20240503 02:13:16.278393 3299 raft_consensus.cc:504] T c87e7cbed0e843f5b5bb9bddef598cdb P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:16.279179 3299 leader_election.cc:288] T c87e7cbed0e843f5b5bb9bddef598cdb P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Requested vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:13:16.280208 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c87e7cbed0e843f5b5bb9bddef598cdb" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:13:16.280272 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c87e7cbed0e843f5b5bb9bddef598cdb" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:13:16.280413 30134 raft_consensus.cc:3037] T c87e7cbed0e843f5b5bb9bddef598cdb P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:16.280637 567 raft_consensus.cc:3037] T c87e7cbed0e843f5b5bb9bddef598cdb P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:16.283735 30134 raft_consensus.cc:2454] T c87e7cbed0e843f5b5bb9bddef598cdb P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:13:16.285806 29729 catalog_manager.cc:5576] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233) does not have the latest schema for tablet d66a33b0d22142b19b502dfedd6c4cdd (table rebalancer_dynamic_table [id=680497bb39cf42e9aa4264c5ace9ca85]). Expected version 1 got 0
I20240503 02:13:16.286811 32708 leader_election.cc:302] T c87e7cbed0e843f5b5bb9bddef598cdb P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:13:16.288614 567 raft_consensus.cc:2454] T c87e7cbed0e843f5b5bb9bddef598cdb P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:13:16.289021 3299 raft_consensus.cc:2781] T c87e7cbed0e843f5b5bb9bddef598cdb P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:16.289757 3299 raft_consensus.cc:686] T c87e7cbed0e843f5b5bb9bddef598cdb P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Becoming Leader. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Running, Role: LEADER
I20240503 02:13:16.290903 3299 consensus_queue.cc:237] T c87e7cbed0e843f5b5bb9bddef598cdb P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:16.298519 3320 tablet.cc:1722] T d66a33b0d22142b19b502dfedd6c4cdd P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:16.300540 3320 tablet.cc:1620] T d66a33b0d22142b19b502dfedd6c4cdd P f575ce92c0224eeb80b9fae5f6561fbe: MemRowSet was empty: no flush needed.
I20240503 02:13:16.301379 29725 catalog_manager.cc:5576] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233) does not have the latest schema for tablet d66a33b0d22142b19b502dfedd6c4cdd (table rebalancer_dynamic_table [id=680497bb39cf42e9aa4264c5ace9ca85]). Expected version 1 got 0
I20240503 02:13:16.304704 29725 catalog_manager.cc:5484] T c87e7cbed0e843f5b5bb9bddef598cdb P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6). New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:16.313731 30134 consensus_queue.cc:237] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:16.316054 32640 raft_consensus.cc:1261] T 7076453cbe5944eb9b075127e1e0495d P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:16.316112 30006 raft_consensus.cc:1261] T 7076453cbe5944eb9b075127e1e0495d P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:16.316550 2773 consensus_queue.cc:1035] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:16.316859 3266 consensus_queue.cc:1035] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:16.319656 3266 raft_consensus.cc:2932] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:16.321404 30006 raft_consensus.cc:2932] T 7076453cbe5944eb9b075127e1e0495d P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:16.328431 29729 catalog_manager.cc:5484] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 reported cstate change: config changed from index 4 to 5, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:16.332124 29853 tablet_service.cc:1514] Processing DeleteTablet for tablet 7076453cbe5944eb9b075127e1e0495d with delete_type TABLET_DATA_TOMBSTONED (TS 16440e365430405fb10366ab70e2b429 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:58744
I20240503 02:13:16.333102 3329 tablet_replica.cc:330] T 7076453cbe5944eb9b075127e1e0495d P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:13:16.339522 32641 raft_consensus.cc:2932] T 7076453cbe5944eb9b075127e1e0495d P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:16.341363 3329 raft_consensus.cc:2229] T 7076453cbe5944eb9b075127e1e0495d P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:16.342058 3329 raft_consensus.cc:2258] T 7076453cbe5944eb9b075127e1e0495d P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:16.342813 3329 ts_tablet_manager.cc:1875] T 7076453cbe5944eb9b075127e1e0495d P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:16.351439 3329 ts_tablet_manager.cc:1888] T 7076453cbe5944eb9b075127e1e0495d P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:13:16.351580 3329 log.cc:1196] T 7076453cbe5944eb9b075127e1e0495d P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/7076453cbe5944eb9b075127e1e0495d
I20240503 02:13:16.352607 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet 7076453cbe5944eb9b075127e1e0495d (table rebalance_test_table_9 [id=e80de094521a4d71bc2fb5f3343275b8]) successfully deleted
I20240503 02:13:16.393862 29729 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:13:16.408744 3312 tablet.cc:1722] T d66a33b0d22142b19b502dfedd6c4cdd P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:16.409271 3312 tablet.cc:1620] T d66a33b0d22142b19b502dfedd6c4cdd P fbbc2be6382e47908ecb7c68c367bd26: MemRowSet was empty: no flush needed.
I20240503 02:13:16.409862 3313 tablet.cc:1722] T d66a33b0d22142b19b502dfedd6c4cdd P a9094240860e43bfb129193f3bfd5be6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:16.410315 3313 tablet.cc:1620] T d66a33b0d22142b19b502dfedd6c4cdd P a9094240860e43bfb129193f3bfd5be6: MemRowSet was empty: no flush needed.
I20240503 02:13:16.414469 3320 tablet.cc:1722] T d66a33b0d22142b19b502dfedd6c4cdd P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:16.414952 3320 tablet.cc:1620] T d66a33b0d22142b19b502dfedd6c4cdd P f575ce92c0224eeb80b9fae5f6561fbe: MemRowSet was empty: no flush needed.
I20240503 02:13:16.424777 29729 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=680497bb39cf42e9aa4264c5ace9ca85] alter complete (version 2)
I20240503 02:13:16.540284 29734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:13:16.549711 3312 tablet.cc:1722] T d66a33b0d22142b19b502dfedd6c4cdd P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:16.553200 3320 tablet.cc:1722] T d66a33b0d22142b19b502dfedd6c4cdd P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:16.565685 3313 tablet.cc:1722] T d66a33b0d22142b19b502dfedd6c4cdd P a9094240860e43bfb129193f3bfd5be6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:16.571241 29725 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table_new_name [id=680497bb39cf42e9aa4264c5ace9ca85] alter complete (version 3)
I20240503 02:13:16.664129 29984 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:13:16.678292 29729 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:16.678602 29729 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:16.682390 3299 consensus_queue.cc:1035] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:16.689181 29729 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet d66a33b0d22142b19b502dfedd6c4cdd
I20240503 02:13:16.690734 547 tablet_service.cc:1514] Processing DeleteTablet for tablet d66a33b0d22142b19b502dfedd6c4cdd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:16 UTC) from {username='slave'} at 127.0.0.1:39028
I20240503 02:13:16.690930 416 tablet_service.cc:1514] Processing DeleteTablet for tablet d66a33b0d22142b19b502dfedd6c4cdd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:16 UTC) from {username='slave'} at 127.0.0.1:50814
I20240503 02:13:16.692394 30116 tablet_service.cc:1514] Processing DeleteTablet for tablet d66a33b0d22142b19b502dfedd6c4cdd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:16 UTC) from {username='slave'} at 127.0.0.1:44812
I20240503 02:13:16.694387 2773 consensus_queue.cc:1035] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:16.706024 3271 consensus_queue.cc:1035] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:16.731408 2931 consensus_queue.cc:1035] T c87e7cbed0e843f5b5bb9bddef598cdb P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:16.737022 29853 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:13:16.761221 3387 tablet_replica.cc:330] T d66a33b0d22142b19b502dfedd6c4cdd P fbbc2be6382e47908ecb7c68c367bd26: stopping tablet replica
I20240503 02:13:16.761461 3387 raft_consensus.cc:2229] T d66a33b0d22142b19b502dfedd6c4cdd P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:16.761766 3387 raft_consensus.cc:2258] T d66a33b0d22142b19b502dfedd6c4cdd P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:16.762868 3387 ts_tablet_manager.cc:1875] T d66a33b0d22142b19b502dfedd6c4cdd P fbbc2be6382e47908ecb7c68c367bd26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:16.763132 3390 tablet_replica.cc:330] T d66a33b0d22142b19b502dfedd6c4cdd P a9094240860e43bfb129193f3bfd5be6: stopping tablet replica
I20240503 02:13:16.763345 3390 raft_consensus.cc:2229] T d66a33b0d22142b19b502dfedd6c4cdd P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:16.763646 3390 raft_consensus.cc:2258] T d66a33b0d22142b19b502dfedd6c4cdd P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:16.778817 2931 consensus_queue.cc:1035] T c87e7cbed0e843f5b5bb9bddef598cdb P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:16.782063 2931 consensus_queue.cc:1035] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:16.792343 29729 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:13:16.793493 3390 ts_tablet_manager.cc:1875] T d66a33b0d22142b19b502dfedd6c4cdd P a9094240860e43bfb129193f3bfd5be6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:16.804980 30245 tablet_service.cc:1467] Processing CreateTablet for tablet f5370f4730794e72806446e1e71d7c5f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d081257b79bd4fc9beade07061b17fba]), partition=
I20240503 02:13:16.805527 30245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5370f4730794e72806446e1e71d7c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:16.812546 546 tablet_service.cc:1467] Processing CreateTablet for tablet f5370f4730794e72806446e1e71d7c5f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d081257b79bd4fc9beade07061b17fba]), partition=
I20240503 02:13:16.812656 32619 tablet_service.cc:1467] Processing CreateTablet for tablet f5370f4730794e72806446e1e71d7c5f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d081257b79bd4fc9beade07061b17fba]), partition=
I20240503 02:13:16.813076 32619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5370f4730794e72806446e1e71d7c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:16.813083 546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5370f4730794e72806446e1e71d7c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:16.812114 3386 tablet_replica.cc:330] T d66a33b0d22142b19b502dfedd6c4cdd P f575ce92c0224eeb80b9fae5f6561fbe: stopping tablet replica
I20240503 02:13:16.827024 3386 raft_consensus.cc:2229] T d66a33b0d22142b19b502dfedd6c4cdd P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:16.827675 3386 raft_consensus.cc:2258] T d66a33b0d22142b19b502dfedd6c4cdd P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:16.828626 3386 ts_tablet_manager.cc:1875] T d66a33b0d22142b19b502dfedd6c4cdd P f575ce92c0224eeb80b9fae5f6561fbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:16.843644 3387 ts_tablet_manager.cc:1888] T d66a33b0d22142b19b502dfedd6c4cdd P fbbc2be6382e47908ecb7c68c367bd26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:16.849453 32621 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:16.850612 3387 log.cc:1196] T d66a33b0d22142b19b502dfedd6c4cdd P fbbc2be6382e47908ecb7c68c367bd26: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal/wals/d66a33b0d22142b19b502dfedd6c4cdd
I20240503 02:13:16.853518 30116 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:16.863608 30244 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:16.868503 3397 tablet_bootstrap.cc:492] T f5370f4730794e72806446e1e71d7c5f P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:16.874770 29725 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_extra_table_40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:16.894165 29853 tablet_service.cc:1467] Processing CreateTablet for tablet 493c813acc27492fb952791cf2dc12a2 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=2285665685314d5cae1699d9f2c791f0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:16.894310 29984 tablet_service.cc:1467] Processing CreateTablet for tablet 493c813acc27492fb952791cf2dc12a2 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=2285665685314d5cae1699d9f2c791f0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:16.894634 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493c813acc27492fb952791cf2dc12a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:16.894673 29984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493c813acc27492fb952791cf2dc12a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:16.898639 30244 tablet_service.cc:1467] Processing CreateTablet for tablet 89938d4c793a44408795398078cc2ee0 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=2285665685314d5cae1699d9f2c791f0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:16.899061 30244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89938d4c793a44408795398078cc2ee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:16.902240 29986 tablet_service.cc:1467] Processing CreateTablet for tablet 89938d4c793a44408795398078cc2ee0 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=2285665685314d5cae1699d9f2c791f0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:16.902613 29986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89938d4c793a44408795398078cc2ee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:16.906035 29854 tablet_service.cc:1467] Processing CreateTablet for tablet 89938d4c793a44408795398078cc2ee0 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=2285665685314d5cae1699d9f2c791f0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:16.906389 29854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89938d4c793a44408795398078cc2ee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:16.912375 29985 tablet_service.cc:1467] Processing CreateTablet for tablet 5a5d5658fe5e4104a15b404cb2022b7c (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=2285665685314d5cae1699d9f2c791f0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:16.912761 29985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a5d5658fe5e4104a15b404cb2022b7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:16.916520 3390 ts_tablet_manager.cc:1888] T d66a33b0d22142b19b502dfedd6c4cdd P a9094240860e43bfb129193f3bfd5be6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:16.916676 3390 log.cc:1196] T d66a33b0d22142b19b502dfedd6c4cdd P a9094240860e43bfb129193f3bfd5be6: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/wals/d66a33b0d22142b19b502dfedd6c4cdd
I20240503 02:13:16.917162 3390 ts_tablet_manager.cc:1909] T d66a33b0d22142b19b502dfedd6c4cdd P a9094240860e43bfb129193f3bfd5be6: Deleting consensus metadata
I20240503 02:13:16.918277 29718 catalog_manager.cc:4830] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): tablet d66a33b0d22142b19b502dfedd6c4cdd (table rebalancer_dynamic_table_new_name [id=680497bb39cf42e9aa4264c5ace9ca85]) successfully deleted
I20240503 02:13:16.929888 30116 tablet_service.cc:1467] Processing CreateTablet for tablet 5a5d5658fe5e4104a15b404cb2022b7c (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=2285665685314d5cae1699d9f2c791f0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:16.930362 30116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a5d5658fe5e4104a15b404cb2022b7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:16.955628 3400 tablet_bootstrap.cc:492] T f5370f4730794e72806446e1e71d7c5f P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:13:16.960379 3401 tablet_bootstrap.cc:492] T f5370f4730794e72806446e1e71d7c5f P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:16.962646 3401 tablet_bootstrap.cc:654] T f5370f4730794e72806446e1e71d7c5f P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:16.969820 3409 tablet_bootstrap.cc:492] T 89938d4c793a44408795398078cc2ee0 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:13:16.970924 3401 tablet_bootstrap.cc:492] T f5370f4730794e72806446e1e71d7c5f P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:16.973210 3406 tablet_bootstrap.cc:492] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:13:16.975800 3406 tablet_bootstrap.cc:654] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:16.983258 3400 tablet_bootstrap.cc:654] T f5370f4730794e72806446e1e71d7c5f P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:16.986347 3387 ts_tablet_manager.cc:1909] T d66a33b0d22142b19b502dfedd6c4cdd P fbbc2be6382e47908ecb7c68c367bd26: Deleting consensus metadata
I20240503 02:13:16.987555 29719 catalog_manager.cc:4830] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): tablet d66a33b0d22142b19b502dfedd6c4cdd (table rebalancer_dynamic_table_new_name [id=680497bb39cf42e9aa4264c5ace9ca85]) successfully deleted
I20240503 02:13:16.889580 546 tablet_service.cc:1467] Processing CreateTablet for tablet 493c813acc27492fb952791cf2dc12a2 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=2285665685314d5cae1699d9f2c791f0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:16.999707 546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493c813acc27492fb952791cf2dc12a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:17.001188 3401 ts_tablet_manager.cc:1367] T f5370f4730794e72806446e1e71d7c5f P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.000s
I20240503 02:13:17.002050 3397 tablet_bootstrap.cc:654] T f5370f4730794e72806446e1e71d7c5f P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:17.002120 3401 raft_consensus.cc:348] T f5370f4730794e72806446e1e71d7c5f P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:17.002383 3401 raft_consensus.cc:374] T f5370f4730794e72806446e1e71d7c5f P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:17.002498 3401 raft_consensus.cc:729] T f5370f4730794e72806446e1e71d7c5f P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:17.002794 3401 consensus_queue.cc:260] T f5370f4730794e72806446e1e71d7c5f P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:17.004184 3401 ts_tablet_manager.cc:1398] T f5370f4730794e72806446e1e71d7c5f P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:17.010758 547 tablet_service.cc:1467] Processing CreateTablet for tablet 5a5d5658fe5e4104a15b404cb2022b7c (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=2285665685314d5cae1699d9f2c791f0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:17.011159 547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a5d5658fe5e4104a15b404cb2022b7c. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:13:17.012612 32708 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.3:42679, user_credentials={real_user=slave}} blocked reactor thread for 125602us
I20240503 02:13:16.925644 3386 ts_tablet_manager.cc:1888] T d66a33b0d22142b19b502dfedd6c4cdd P f575ce92c0224eeb80b9fae5f6561fbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:17.024669 3386 log.cc:1196] T d66a33b0d22142b19b502dfedd6c4cdd P f575ce92c0224eeb80b9fae5f6561fbe: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal/wals/d66a33b0d22142b19b502dfedd6c4cdd
I20240503 02:13:17.025975 3406 tablet_bootstrap.cc:492] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:13:17.026180 3406 ts_tablet_manager.cc:1367] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.053s user 0.005s sys 0.014s
I20240503 02:13:17.027343 3406 raft_consensus.cc:348] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:17.027736 3386 ts_tablet_manager.cc:1909] T d66a33b0d22142b19b502dfedd6c4cdd P f575ce92c0224eeb80b9fae5f6561fbe: Deleting consensus metadata
I20240503 02:13:17.027621 3406 raft_consensus.cc:374] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:17.028648 3406 raft_consensus.cc:729] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:13:17.030680 3406 consensus_queue.cc:260] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:17.031431 3406 ts_tablet_manager.cc:1398] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240503 02:13:17.034214 3409 tablet_bootstrap.cc:654] T 89938d4c793a44408795398078cc2ee0 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:17.036216 3406 tablet_bootstrap.cc:492] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:13:17.041293 3400 tablet_bootstrap.cc:492] T f5370f4730794e72806446e1e71d7c5f P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:13:17.041501 3400 ts_tablet_manager.cc:1367] T f5370f4730794e72806446e1e71d7c5f P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.086s user 0.004s sys 0.000s
I20240503 02:13:17.042390 3400 raft_consensus.cc:348] T f5370f4730794e72806446e1e71d7c5f P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:17.042676 3400 raft_consensus.cc:374] T f5370f4730794e72806446e1e71d7c5f P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:17.042798 3400 raft_consensus.cc:729] T f5370f4730794e72806446e1e71d7c5f P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:13:17.043085 3400 consensus_queue.cc:260] T f5370f4730794e72806446e1e71d7c5f P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:17.044550 29718 catalog_manager.cc:4830] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): tablet d66a33b0d22142b19b502dfedd6c4cdd (table rebalancer_dynamic_table_new_name [id=680497bb39cf42e9aa4264c5ace9ca85]) successfully deleted
I20240503 02:13:17.053773 3406 tablet_bootstrap.cc:654] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:17.060995 3409 tablet_bootstrap.cc:492] T 89938d4c793a44408795398078cc2ee0 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:13:17.061269 3409 ts_tablet_manager.cc:1367] T 89938d4c793a44408795398078cc2ee0 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.092s user 0.003s sys 0.001s
I20240503 02:13:17.061414 546 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:17.062183 3406 tablet_bootstrap.cc:492] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:13:17.062067 3409 raft_consensus.cc:348] T 89938d4c793a44408795398078cc2ee0 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:17.062247 3409 raft_consensus.cc:374] T 89938d4c793a44408795398078cc2ee0 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:17.062417 3406 ts_tablet_manager.cc:1367] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.007s
I20240503 02:13:17.062799 3400 ts_tablet_manager.cc:1398] T f5370f4730794e72806446e1e71d7c5f P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.021s user 0.001s sys 0.000s
I20240503 02:13:17.063163 3406 raft_consensus.cc:348] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:17.063454 3406 raft_consensus.cc:374] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:17.063561 3406 raft_consensus.cc:729] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:13:17.063853 3406 consensus_queue.cc:260] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:17.064641 3406 ts_tablet_manager.cc:1398] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:13:17.065333 3400 tablet_bootstrap.cc:492] T 89938d4c793a44408795398078cc2ee0 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:13:17.069535 3397 tablet_bootstrap.cc:492] T f5370f4730794e72806446e1e71d7c5f P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:17.072989 3409 raft_consensus.cc:729] T 89938d4c793a44408795398078cc2ee0 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:13:17.077621 3409 consensus_queue.cc:260] T 89938d4c793a44408795398078cc2ee0 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:17.079071 3400 tablet_bootstrap.cc:654] T 89938d4c793a44408795398078cc2ee0 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:17.079077 32754 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:17.079658 3409 ts_tablet_manager.cc:1398] T 89938d4c793a44408795398078cc2ee0 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.018s user 0.000s sys 0.002s
I20240503 02:13:17.090014 3412 tablet_bootstrap.cc:492] T 5a5d5658fe5e4104a15b404cb2022b7c P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:13:17.091774 3412 tablet_bootstrap.cc:654] T 5a5d5658fe5e4104a15b404cb2022b7c P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:17.097460 3409 tablet_bootstrap.cc:492] T 5a5d5658fe5e4104a15b404cb2022b7c P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:13:17.099431 3409 tablet_bootstrap.cc:654] T 5a5d5658fe5e4104a15b404cb2022b7c P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:17.106618 3409 tablet_bootstrap.cc:492] T 5a5d5658fe5e4104a15b404cb2022b7c P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:13:17.113185 3409 ts_tablet_manager.cc:1367] T 5a5d5658fe5e4104a15b404cb2022b7c P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20240503 02:13:17.113440 3397 ts_tablet_manager.cc:1367] T f5370f4730794e72806446e1e71d7c5f P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.245s user 0.005s sys 0.000s
I20240503 02:13:17.114809 3409 raft_consensus.cc:348] T 5a5d5658fe5e4104a15b404cb2022b7c P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:17.115962 3400 tablet_bootstrap.cc:492] T 89938d4c793a44408795398078cc2ee0 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:13:17.115927 3397 raft_consensus.cc:348] T f5370f4730794e72806446e1e71d7c5f P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:17.116102 3400 ts_tablet_manager.cc:1367] T 89938d4c793a44408795398078cc2ee0 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.000s
I20240503 02:13:17.116140 3397 raft_consensus.cc:374] T f5370f4730794e72806446e1e71d7c5f P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:17.116240 3397 raft_consensus.cc:729] T f5370f4730794e72806446e1e71d7c5f P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:17.116376 3409 raft_consensus.cc:374] T 5a5d5658fe5e4104a15b404cb2022b7c P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:17.116443 3409 raft_consensus.cc:729] T 5a5d5658fe5e4104a15b404cb2022b7c P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:13:17.116638 3409 consensus_queue.cc:260] T 5a5d5658fe5e4104a15b404cb2022b7c P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:17.116825 3400 raft_consensus.cc:348] T 89938d4c793a44408795398078cc2ee0 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:17.117005 3400 raft_consensus.cc:374] T 89938d4c793a44408795398078cc2ee0 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:17.117101 3400 raft_consensus.cc:729] T 89938d4c793a44408795398078cc2ee0 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:13:17.117342 3400 consensus_queue.cc:260] T 89938d4c793a44408795398078cc2ee0 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:17.117587 3397 consensus_queue.cc:260] T f5370f4730794e72806446e1e71d7c5f P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:17.117969 3400 ts_tablet_manager.cc:1398] T 89938d4c793a44408795398078cc2ee0 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:13:17.120575 3409 ts_tablet_manager.cc:1398] T 5a5d5658fe5e4104a15b404cb2022b7c P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 02:13:17.120831 3409 tablet_bootstrap.cc:492] T 493c813acc27492fb952791cf2dc12a2 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:13:17.121807 3397 ts_tablet_manager.cc:1398] T f5370f4730794e72806446e1e71d7c5f P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:13:17.123205 3397 tablet_bootstrap.cc:492] T 5a5d5658fe5e4104a15b404cb2022b7c P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:17.129140 416 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:17.129530 2947 raft_consensus.cc:482] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:17.129638 3412 tablet_bootstrap.cc:492] T 5a5d5658fe5e4104a15b404cb2022b7c P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:13:17.129782 3412 ts_tablet_manager.cc:1367] T 5a5d5658fe5e4104a15b404cb2022b7c P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.000s
I20240503 02:13:17.129678 2947 raft_consensus.cc:504] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:17.130517 2947 leader_election.cc:288] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:13:17.130841 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89938d4c793a44408795398078cc2ee0" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:13:17.130955 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89938d4c793a44408795398078cc2ee0" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:13:17.131038 30266 raft_consensus.cc:2454] T 89938d4c793a44408795398078cc2ee0 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:13:17.131136 30004 raft_consensus.cc:2454] T 89938d4c793a44408795398078cc2ee0 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:13:17.131429 3409 tablet_bootstrap.cc:654] T 493c813acc27492fb952791cf2dc12a2 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:17.132005 29809 leader_election.cc:302] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429; no voters:
I20240503 02:13:17.132275 2947 raft_consensus.cc:2781] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:17.132351 2947 raft_consensus.cc:482] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:17.132408 2947 raft_consensus.cc:3037] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:17.133982 3397 tablet_bootstrap.cc:654] T 5a5d5658fe5e4104a15b404cb2022b7c P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:17.135371 2947 raft_consensus.cc:504] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:17.137796 2947 leader_election.cc:288] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Requested vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:13:17.140156 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89938d4c793a44408795398078cc2ee0" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:13:17.140168 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89938d4c793a44408795398078cc2ee0" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:13:17.140367 30266 raft_consensus.cc:3037] T 89938d4c793a44408795398078cc2ee0 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:17.140372 30004 raft_consensus.cc:3037] T 89938d4c793a44408795398078cc2ee0 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:17.143491 30266 raft_consensus.cc:2454] T 89938d4c793a44408795398078cc2ee0 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:13:17.144687 3397 tablet_bootstrap.cc:492] T 5a5d5658fe5e4104a15b404cb2022b7c P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:17.144834 29809 leader_election.cc:302] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:13:17.145071 3397 ts_tablet_manager.cc:1367] T 5a5d5658fe5e4104a15b404cb2022b7c P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.003s
I20240503 02:13:17.145645 2947 raft_consensus.cc:2781] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:17.145982 2947 raft_consensus.cc:686] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Becoming Leader. State: Replica: 16440e365430405fb10366ab70e2b429, State: Running, Role: LEADER
I20240503 02:13:17.130477 3412 raft_consensus.cc:348] T 5a5d5658fe5e4104a15b404cb2022b7c P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:17.146292 2947 consensus_queue.cc:237] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:17.146636 3412 raft_consensus.cc:374] T 5a5d5658fe5e4104a15b404cb2022b7c P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:17.146787 3412 raft_consensus.cc:729] T 5a5d5658fe5e4104a15b404cb2022b7c P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:13:17.147126 3412 consensus_queue.cc:260] T 5a5d5658fe5e4104a15b404cb2022b7c P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:17.147608 30004 raft_consensus.cc:2454] T 89938d4c793a44408795398078cc2ee0 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:13:17.147876 3412 ts_tablet_manager.cc:1398] T 5a5d5658fe5e4104a15b404cb2022b7c P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240503 02:13:17.152441 3409 tablet_bootstrap.cc:492] T 493c813acc27492fb952791cf2dc12a2 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:13:17.154747 3409 ts_tablet_manager.cc:1367] T 493c813acc27492fb952791cf2dc12a2 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.000s
I20240503 02:13:17.155932 3409 raft_consensus.cc:348] T 493c813acc27492fb952791cf2dc12a2 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:17.156291 3409 raft_consensus.cc:374] T 493c813acc27492fb952791cf2dc12a2 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:17.156755 3409 raft_consensus.cc:729] T 493c813acc27492fb952791cf2dc12a2 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:13:17.157109 3409 consensus_queue.cc:260] T 493c813acc27492fb952791cf2dc12a2 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:17.159960 3409 ts_tablet_manager.cc:1398] T 493c813acc27492fb952791cf2dc12a2 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:13:17.160218 3397 raft_consensus.cc:348] T 5a5d5658fe5e4104a15b404cb2022b7c P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:17.160588 3397 raft_consensus.cc:374] T 5a5d5658fe5e4104a15b404cb2022b7c P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:17.160753 3397 raft_consensus.cc:729] T 5a5d5658fe5e4104a15b404cb2022b7c P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:17.161477 3397 consensus_queue.cc:260] T 5a5d5658fe5e4104a15b404cb2022b7c P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:17.163810 29725 catalog_manager.cc:5484] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16440e365430405fb10366ab70e2b429 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:17.166172 3397 ts_tablet_manager.cc:1398] T 5a5d5658fe5e4104a15b404cb2022b7c P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.018s user 0.000s sys 0.002s
I20240503 02:13:17.166208 2947 raft_consensus.cc:482] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:17.166344 2947 raft_consensus.cc:504] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:17.166985 2947 leader_election.cc:288] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:13:17.167484 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493c813acc27492fb952791cf2dc12a2" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:13:17.167703 30004 raft_consensus.cc:2454] T 493c813acc27492fb952791cf2dc12a2 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:13:17.168309 566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493c813acc27492fb952791cf2dc12a2" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:17.168642 2324 raft_consensus.cc:482] T f5370f4730794e72806446e1e71d7c5f P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:17.168704 29809 leader_election.cc:302] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429; no voters:
I20240503 02:13:17.168795 2324 raft_consensus.cc:504] T f5370f4730794e72806446e1e71d7c5f P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
W20240503 02:13:17.169020 29808 leader_election.cc:341] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:17.169477 3397 tablet_bootstrap.cc:492] T 493c813acc27492fb952791cf2dc12a2 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:17.169543 2947 raft_consensus.cc:2781] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:17.169627 2947 raft_consensus.cc:482] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:17.169709 2947 raft_consensus.cc:3037] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:17.169826 2324 leader_election.cc:288] T f5370f4730794e72806446e1e71d7c5f P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:13:17.170432 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5370f4730794e72806446e1e71d7c5f" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:13:17.170688 32640 raft_consensus.cc:2454] T f5370f4730794e72806446e1e71d7c5f P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:13:17.171334 30200 leader_election.cc:302] T f5370f4730794e72806446e1e71d7c5f P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, 9a4d92bc25374fb3960fde7d10be309d; no voters:
I20240503 02:13:17.171628 2324 raft_consensus.cc:2781] T f5370f4730794e72806446e1e71d7c5f P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:17.171702 2324 raft_consensus.cc:482] T f5370f4730794e72806446e1e71d7c5f P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:17.171758 2324 raft_consensus.cc:3037] T f5370f4730794e72806446e1e71d7c5f P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:17.172240 3397 tablet_bootstrap.cc:654] T 493c813acc27492fb952791cf2dc12a2 P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:17.172734 2947 raft_consensus.cc:504] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:17.174196 2947 leader_election.cc:288] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Requested vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:13:17.174495 2324 raft_consensus.cc:504] T f5370f4730794e72806446e1e71d7c5f P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:17.174758 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493c813acc27492fb952791cf2dc12a2" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:13:17.173905 566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493c813acc27492fb952791cf2dc12a2" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:13:17.174945 30004 raft_consensus.cc:3037] T 493c813acc27492fb952791cf2dc12a2 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:17.175119 2324 leader_election.cc:288] T f5370f4730794e72806446e1e71d7c5f P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Requested vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
W20240503 02:13:17.175280 29808 leader_election.cc:341] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:17.177664 566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5370f4730794e72806446e1e71d7c5f" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:17.177882 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5370f4730794e72806446e1e71d7c5f" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:13:17.177892 566 raft_consensus.cc:2454] T f5370f4730794e72806446e1e71d7c5f P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:13:17.181751 30004 raft_consensus.cc:2454] T 493c813acc27492fb952791cf2dc12a2 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:13:17.181991 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5370f4730794e72806446e1e71d7c5f" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d"
I20240503 02:13:17.182228 32640 raft_consensus.cc:3037] T f5370f4730794e72806446e1e71d7c5f P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:17.185269 29809 leader_election.cc:302] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429; no voters: f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:13:17.187426 32640 raft_consensus.cc:2454] T f5370f4730794e72806446e1e71d7c5f P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:13:17.187819 30200 leader_election.cc:302] T f5370f4730794e72806446e1e71d7c5f P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, 9a4d92bc25374fb3960fde7d10be309d; no voters: f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:13:17.188073 2324 raft_consensus.cc:2781] T f5370f4730794e72806446e1e71d7c5f P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:17.188203 2324 raft_consensus.cc:686] T f5370f4730794e72806446e1e71d7c5f P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Becoming Leader. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Running, Role: LEADER
I20240503 02:13:17.188419 2324 consensus_queue.cc:237] T f5370f4730794e72806446e1e71d7c5f P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:17.190132 2947 raft_consensus.cc:2781] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:17.190450 2947 raft_consensus.cc:686] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Becoming Leader. State: Replica: 16440e365430405fb10366ab70e2b429, State: Running, Role: LEADER
I20240503 02:13:17.190979 2947 consensus_queue.cc:237] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:17.200794 3397 tablet_bootstrap.cc:492] T 493c813acc27492fb952791cf2dc12a2 P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:17.200614 29729 catalog_manager.cc:5484] T f5370f4730794e72806446e1e71d7c5f P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4). New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:17.201114 3397 ts_tablet_manager.cc:1367] T 493c813acc27492fb952791cf2dc12a2 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.003s
I20240503 02:13:17.202129 3397 raft_consensus.cc:348] T 493c813acc27492fb952791cf2dc12a2 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:17.202404 3397 raft_consensus.cc:374] T 493c813acc27492fb952791cf2dc12a2 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:17.202540 3397 raft_consensus.cc:729] T 493c813acc27492fb952791cf2dc12a2 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:17.202853 3397 consensus_queue.cc:260] T 493c813acc27492fb952791cf2dc12a2 P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:17.204985 3397 ts_tablet_manager.cc:1398] T 493c813acc27492fb952791cf2dc12a2 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:13:17.215063 29734 catalog_manager.cc:5484] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16440e365430405fb10366ab70e2b429 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:17.219630 3266 raft_consensus.cc:482] T 5a5d5658fe5e4104a15b404cb2022b7c P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:17.219811 3266 raft_consensus.cc:504] T 5a5d5658fe5e4104a15b404cb2022b7c P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:17.220454 3266 leader_election.cc:288] T 5a5d5658fe5e4104a15b404cb2022b7c P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:13:17.221290 566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a5d5658fe5e4104a15b404cb2022b7c" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:17.221539 566 raft_consensus.cc:2454] T 5a5d5658fe5e4104a15b404cb2022b7c P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 0.
I20240503 02:13:17.222426 30069 leader_election.cc:302] T 5a5d5658fe5e4104a15b404cb2022b7c P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:17.223721 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a5d5658fe5e4104a15b404cb2022b7c" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:13:17.223954 30004 raft_consensus.cc:2454] T 5a5d5658fe5e4104a15b404cb2022b7c P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 0.
I20240503 02:13:17.226189 3266 raft_consensus.cc:2781] T 5a5d5658fe5e4104a15b404cb2022b7c P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:17.226286 3266 raft_consensus.cc:482] T 5a5d5658fe5e4104a15b404cb2022b7c P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:17.226361 3266 raft_consensus.cc:3037] T 5a5d5658fe5e4104a15b404cb2022b7c P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:17.232033 3266 raft_consensus.cc:504] T 5a5d5658fe5e4104a15b404cb2022b7c P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:17.233372 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a5d5658fe5e4104a15b404cb2022b7c" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:13:17.233579 30004 raft_consensus.cc:3037] T 5a5d5658fe5e4104a15b404cb2022b7c P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:17.236268 30004 raft_consensus.cc:2454] T 5a5d5658fe5e4104a15b404cb2022b7c P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 1.
I20240503 02:13:17.237092 30070 leader_election.cc:302] T 5a5d5658fe5e4104a15b404cb2022b7c P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:13:17.238059 3271 raft_consensus.cc:2781] T 5a5d5658fe5e4104a15b404cb2022b7c P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:17.247822 3266 leader_election.cc:288] T 5a5d5658fe5e4104a15b404cb2022b7c P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Requested vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:13:17.266113 566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a5d5658fe5e4104a15b404cb2022b7c" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:13:17.266427 566 raft_consensus.cc:3037] T 5a5d5658fe5e4104a15b404cb2022b7c P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:17.270808 566 raft_consensus.cc:2454] T 5a5d5658fe5e4104a15b404cb2022b7c P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 1.
W20240503 02:13:17.279539 29939 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.3:42679, user_credentials={real_user=slave}} blocked reactor thread for 53874.8us
I20240503 02:13:17.279986 3271 raft_consensus.cc:686] T 5a5d5658fe5e4104a15b404cb2022b7c P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Becoming Leader. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Running, Role: LEADER
I20240503 02:13:17.280647 3271 consensus_queue.cc:237] T 5a5d5658fe5e4104a15b404cb2022b7c P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:17.285452 29734 catalog_manager.cc:5484] T 5a5d5658fe5e4104a15b404cb2022b7c P a9094240860e43bfb129193f3bfd5be6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9094240860e43bfb129193f3bfd5be6 (127.22.108.3). New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:17.323832 30004 consensus_queue.cc:237] T 08ee37b160d441ddabb640be907d6771 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:17.326047 307 raft_consensus.cc:1261] T 08ee37b160d441ddabb640be907d6771 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:17.326071 30266 raft_consensus.cc:1261] T 08ee37b160d441ddabb640be907d6771 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:17.326546 3431 consensus_queue.cc:1035] T 08ee37b160d441ddabb640be907d6771 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:17.326807 2942 consensus_queue.cc:1035] T 08ee37b160d441ddabb640be907d6771 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:17.329355 3431 raft_consensus.cc:2932] T 08ee37b160d441ddabb640be907d6771 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:17.330161 30266 raft_consensus.cc:2932] T 08ee37b160d441ddabb640be907d6771 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:17.330330 307 raft_consensus.cc:2932] T 08ee37b160d441ddabb640be907d6771 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:17.336545 29729 catalog_manager.cc:5484] T 08ee37b160d441ddabb640be907d6771 P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:17.341048 30116 tablet_service.cc:1514] Processing DeleteTablet for tablet 08ee37b160d441ddabb640be907d6771 with delete_type TABLET_DATA_TOMBSTONED (TS a9094240860e43bfb129193f3bfd5be6 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44812
I20240503 02:13:17.341506 3390 tablet_replica.cc:330] T 08ee37b160d441ddabb640be907d6771 P a9094240860e43bfb129193f3bfd5be6: stopping tablet replica
I20240503 02:13:17.341755 3390 raft_consensus.cc:2229] T 08ee37b160d441ddabb640be907d6771 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:17.342087 3390 raft_consensus.cc:2258] T 08ee37b160d441ddabb640be907d6771 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:17.342727 3390 ts_tablet_manager.cc:1875] T 08ee37b160d441ddabb640be907d6771 P a9094240860e43bfb129193f3bfd5be6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:17.349020 3390 ts_tablet_manager.cc:1888] T 08ee37b160d441ddabb640be907d6771 P a9094240860e43bfb129193f3bfd5be6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:13:17.351110 3390 log.cc:1196] T 08ee37b160d441ddabb640be907d6771 P a9094240860e43bfb129193f3bfd5be6: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/wals/08ee37b160d441ddabb640be907d6771
I20240503 02:13:17.352031 29718 catalog_manager.cc:4830] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): tablet 08ee37b160d441ddabb640be907d6771 (table rebalance_test_table_19 [id=3c36e49b52c04307a655de366327db8c]) successfully deleted
I20240503 02:13:17.374055 29734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:13:17.380479 566 raft_consensus.cc:3037] T f5370f4730794e72806446e1e71d7c5f P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:17.380502 32640 raft_consensus.cc:1261] T f5370f4730794e72806446e1e71d7c5f P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:17.381083 3289 consensus_queue.cc:1035] T f5370f4730794e72806446e1e71d7c5f P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:17.383714 566 raft_consensus.cc:1261] T f5370f4730794e72806446e1e71d7c5f P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:17.384187 3289 consensus_queue.cc:1035] T f5370f4730794e72806446e1e71d7c5f P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:17.404464 29734 catalog_manager.cc:5576] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233) does not have the latest schema for tablet f5370f4730794e72806446e1e71d7c5f (table rebalancer_dynamic_table [id=d081257b79bd4fc9beade07061b17fba]). Expected version 1 got 0
I20240503 02:13:17.410161 3472 tablet.cc:1722] T f5370f4730794e72806446e1e71d7c5f P 1f10f80f13c84e80b03eda7085a3b012: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:17.410847 3472 tablet.cc:1620] T f5370f4730794e72806446e1e71d7c5f P 1f10f80f13c84e80b03eda7085a3b012: MemRowSet was empty: no flush needed.
I20240503 02:13:17.419427 29734 catalog_manager.cc:5576] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595) does not have the latest schema for tablet f5370f4730794e72806446e1e71d7c5f (table rebalancer_dynamic_table [id=d081257b79bd4fc9beade07061b17fba]). Expected version 1 got 0
I20240503 02:13:17.419503 3474 tablet.cc:1722] T f5370f4730794e72806446e1e71d7c5f P 9a4d92bc25374fb3960fde7d10be309d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:17.420228 3474 tablet.cc:1620] T f5370f4730794e72806446e1e71d7c5f P 9a4d92bc25374fb3960fde7d10be309d: MemRowSet was empty: no flush needed.
I20240503 02:13:17.424137 3473 tablet.cc:1722] T f5370f4730794e72806446e1e71d7c5f P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:17.426750 3473 tablet.cc:1620] T f5370f4730794e72806446e1e71d7c5f P f575ce92c0224eeb80b9fae5f6561fbe: MemRowSet was empty: no flush needed.
I20240503 02:13:17.427983 29734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=d081257b79bd4fc9beade07061b17fba] alter complete (version 1)
I20240503 02:13:17.440287 29734 catalog_manager.cc:5576] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233) does not have the latest schema for tablet f5370f4730794e72806446e1e71d7c5f (table rebalancer_dynamic_table [id=d081257b79bd4fc9beade07061b17fba]). Expected version 1 got 0
I20240503 02:13:17.467978 3453 raft_consensus.cc:482] T 493c813acc27492fb952791cf2dc12a2 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:17.468175 3453 raft_consensus.cc:504] T 493c813acc27492fb952791cf2dc12a2 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:17.469002 3453 leader_election.cc:288] T 493c813acc27492fb952791cf2dc12a2 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:13:17.469609 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493c813acc27492fb952791cf2dc12a2" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:13:17.469871 30004 raft_consensus.cc:2379] T 493c813acc27492fb952791cf2dc12a2 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f575ce92c0224eeb80b9fae5f6561fbe in current term 1: Already voted for candidate 16440e365430405fb10366ab70e2b429 in this term.
I20240503 02:13:17.469784 29874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493c813acc27492fb952791cf2dc12a2" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:13:17.470538 504 leader_election.cc:302] T 493c813acc27492fb952791cf2dc12a2 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f575ce92c0224eeb80b9fae5f6561fbe; no voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429
I20240503 02:13:17.471153 3453 raft_consensus.cc:3037] T 493c813acc27492fb952791cf2dc12a2 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:17.474673 3453 raft_consensus.cc:2726] T 493c813acc27492fb952791cf2dc12a2 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:17.528836 2947 consensus_queue.cc:1035] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:17.532685 29725 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:13:17.542537 3472 tablet.cc:1722] T f5370f4730794e72806446e1e71d7c5f P 1f10f80f13c84e80b03eda7085a3b012: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:17.543033 3472 tablet.cc:1620] T f5370f4730794e72806446e1e71d7c5f P 1f10f80f13c84e80b03eda7085a3b012: MemRowSet was empty: no flush needed.
I20240503 02:13:17.549209 3473 tablet.cc:1722] T f5370f4730794e72806446e1e71d7c5f P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:17.549813 3473 tablet.cc:1620] T f5370f4730794e72806446e1e71d7c5f P f575ce92c0224eeb80b9fae5f6561fbe: MemRowSet was empty: no flush needed.
I20240503 02:13:17.552268 3474 tablet.cc:1722] T f5370f4730794e72806446e1e71d7c5f P 9a4d92bc25374fb3960fde7d10be309d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:17.552783 3474 tablet.cc:1620] T f5370f4730794e72806446e1e71d7c5f P 9a4d92bc25374fb3960fde7d10be309d: MemRowSet was empty: no flush needed.
I20240503 02:13:17.560644 29731 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=d081257b79bd4fc9beade07061b17fba] alter complete (version 2)
I20240503 02:13:17.584293 2755 consensus_queue.cc:1035] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:17.664769 29729 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:13:17.674114 3472 tablet.cc:1722] T f5370f4730794e72806446e1e71d7c5f P 1f10f80f13c84e80b03eda7085a3b012: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:17.674965 3474 tablet.cc:1722] T f5370f4730794e72806446e1e71d7c5f P 9a4d92bc25374fb3960fde7d10be309d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:17.681936 3473 tablet.cc:1722] T f5370f4730794e72806446e1e71d7c5f P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:17.689576 29730 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table_new_name [id=d081257b79bd4fc9beade07061b17fba] alter complete (version 3)
I20240503 02:13:17.692651 2947 consensus_queue.cc:1035] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:17.699882 2755 consensus_queue.cc:1035] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:17.712543 29986 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:13:17.731333 29853 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:17.752413 30245 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:13:17.775472 3433 consensus_queue.cc:1035] T 5a5d5658fe5e4104a15b404cb2022b7c P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:17.793298 29730 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:17.793354 29729 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_extra_table_42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:17.793503 29730 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:17.796959 32619 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:17.805766 29730 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet f5370f4730794e72806446e1e71d7c5f
I20240503 02:13:17.806207 32754 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:17.806962 30245 tablet_service.cc:1514] Processing DeleteTablet for tablet f5370f4730794e72806446e1e71d7c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:17 UTC) from {username='slave'} at 127.0.0.1:49856
I20240503 02:13:17.807111 546 tablet_service.cc:1514] Processing DeleteTablet for tablet f5370f4730794e72806446e1e71d7c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:17 UTC) from {username='slave'} at 127.0.0.1:39028
I20240503 02:13:17.819077 3271 consensus_queue.cc:1035] T 5a5d5658fe5e4104a15b404cb2022b7c P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:17.842017 29986 tablet_service.cc:1467] Processing CreateTablet for tablet 1758ec3783b24d3c820ed0d6dc118e69 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=b579d0acbdab49ba82f3db319ec150ce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:17.842558 29986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1758ec3783b24d3c820ed0d6dc118e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:17.842645 29853 tablet_service.cc:1467] Processing CreateTablet for tablet 20019d63ee3e45418dc9c99bd25b72af (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=b579d0acbdab49ba82f3db319ec150ce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:17.843012 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20019d63ee3e45418dc9c99bd25b72af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:17.844724 3528 tablet_replica.cc:330] T f5370f4730794e72806446e1e71d7c5f P 1f10f80f13c84e80b03eda7085a3b012: stopping tablet replica
I20240503 02:13:17.845736 3528 raft_consensus.cc:2229] T f5370f4730794e72806446e1e71d7c5f P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:17.846328 416 tablet_service.cc:1467] Processing CreateTablet for tablet 20019d63ee3e45418dc9c99bd25b72af (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=b579d0acbdab49ba82f3db319ec150ce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:17.846786 3528 raft_consensus.cc:2258] T f5370f4730794e72806446e1e71d7c5f P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:17.846819 416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20019d63ee3e45418dc9c99bd25b72af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:17.849391 417 tablet_service.cc:1467] Processing CreateTablet for tablet 1758ec3783b24d3c820ed0d6dc118e69 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=b579d0acbdab49ba82f3db319ec150ce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:17.849838 417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1758ec3783b24d3c820ed0d6dc118e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:17.852661 415 tablet_service.cc:1467] Processing CreateTablet for tablet eec091fa0ba947d68577d562e538ab56 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=b579d0acbdab49ba82f3db319ec150ce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:17.853406 29854 tablet_service.cc:1467] Processing CreateTablet for tablet 1758ec3783b24d3c820ed0d6dc118e69 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=b579d0acbdab49ba82f3db319ec150ce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:17.853826 29854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1758ec3783b24d3c820ed0d6dc118e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:17.855120 3528 ts_tablet_manager.cc:1875] T f5370f4730794e72806446e1e71d7c5f P 1f10f80f13c84e80b03eda7085a3b012: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:17.855958 32619 tablet_service.cc:1514] Processing DeleteTablet for tablet f5370f4730794e72806446e1e71d7c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:17 UTC) from {username='slave'} at 127.0.0.1:38200
I20240503 02:13:17.856364 32621 tablet_service.cc:1467] Processing CreateTablet for tablet eec091fa0ba947d68577d562e538ab56 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=b579d0acbdab49ba82f3db319ec150ce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:17.856904 32621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eec091fa0ba947d68577d562e538ab56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:17.863976 415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eec091fa0ba947d68577d562e538ab56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:17.864439 30116 tablet_service.cc:1467] Processing CreateTablet for tablet 20019d63ee3e45418dc9c99bd25b72af (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=b579d0acbdab49ba82f3db319ec150ce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:17.864923 30116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20019d63ee3e45418dc9c99bd25b72af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:17.869482 547 tablet_service.cc:1467] Processing CreateTablet for tablet eec091fa0ba947d68577d562e538ab56 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=b579d0acbdab49ba82f3db319ec150ce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:17.870012 547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eec091fa0ba947d68577d562e538ab56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:17.902037 3533 tablet_bootstrap.cc:492] T 20019d63ee3e45418dc9c99bd25b72af P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:13:17.903070 3534 tablet_replica.cc:330] T f5370f4730794e72806446e1e71d7c5f P 9a4d92bc25374fb3960fde7d10be309d: stopping tablet replica
I20240503 02:13:17.903461 3534 raft_consensus.cc:2229] T f5370f4730794e72806446e1e71d7c5f P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:17.904026 3534 raft_consensus.cc:2258] T f5370f4730794e72806446e1e71d7c5f P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:17.907688 3534 ts_tablet_manager.cc:1875] T f5370f4730794e72806446e1e71d7c5f P 9a4d92bc25374fb3960fde7d10be309d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:17.914042 3536 tablet_bootstrap.cc:492] T eec091fa0ba947d68577d562e538ab56 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:17.916014 29729 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:13:17.923554 3541 tablet_bootstrap.cc:492] T 1758ec3783b24d3c820ed0d6dc118e69 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:17.926429 3541 tablet_bootstrap.cc:654] T 1758ec3783b24d3c820ed0d6dc118e69 P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:17.942611 3533 tablet_bootstrap.cc:654] T 20019d63ee3e45418dc9c99bd25b72af P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:17.944061 3536 tablet_bootstrap.cc:654] T eec091fa0ba947d68577d562e538ab56 P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:17.947644 3539 tablet_bootstrap.cc:492] T 20019d63ee3e45418dc9c99bd25b72af P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:13:17.948108 416 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:17.949661 3529 tablet_replica.cc:330] T f5370f4730794e72806446e1e71d7c5f P f575ce92c0224eeb80b9fae5f6561fbe: stopping tablet replica
I20240503 02:13:17.950044 3529 raft_consensus.cc:2229] T f5370f4730794e72806446e1e71d7c5f P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:17.950805 3529 raft_consensus.cc:2258] T f5370f4730794e72806446e1e71d7c5f P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:17.952208 3532 tablet_bootstrap.cc:492] T 1758ec3783b24d3c820ed0d6dc118e69 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:13:17.954053 3532 tablet_bootstrap.cc:654] T 1758ec3783b24d3c820ed0d6dc118e69 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:17.955164 3538 tablet_bootstrap.cc:492] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:17.957986 3538 tablet_bootstrap.cc:654] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:17.968343 30245 tablet_service.cc:1467] Processing CreateTablet for tablet 44a17dadf4d84ae8b18fc12233ac6c81 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5a4aa82bed34461680c15babb4e47e3b]), partition=
I20240503 02:13:17.968735 30245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44a17dadf4d84ae8b18fc12233ac6c81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:17.969193 29853 tablet_service.cc:1467] Processing CreateTablet for tablet 44a17dadf4d84ae8b18fc12233ac6c81 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5a4aa82bed34461680c15babb4e47e3b]), partition=
I20240503 02:13:17.969488 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44a17dadf4d84ae8b18fc12233ac6c81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:17.969785 30116 tablet_service.cc:1467] Processing CreateTablet for tablet 44a17dadf4d84ae8b18fc12233ac6c81 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5a4aa82bed34461680c15babb4e47e3b]), partition=
I20240503 02:13:17.970153 30116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44a17dadf4d84ae8b18fc12233ac6c81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:17.984138 3545 tablet_bootstrap.cc:492] T 44a17dadf4d84ae8b18fc12233ac6c81 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:13:17.986752 3545 tablet_bootstrap.cc:654] T 44a17dadf4d84ae8b18fc12233ac6c81 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:17.988230 3539 tablet_bootstrap.cc:654] T 20019d63ee3e45418dc9c99bd25b72af P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:17.994535 30114 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:13:17.995685 3545 tablet_bootstrap.cc:492] T 44a17dadf4d84ae8b18fc12233ac6c81 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:13:17.996271 3545 ts_tablet_manager.cc:1367] T 44a17dadf4d84ae8b18fc12233ac6c81 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20240503 02:13:17.997398 3545 raft_consensus.cc:348] T 44a17dadf4d84ae8b18fc12233ac6c81 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:17.998850 3545 raft_consensus.cc:374] T 44a17dadf4d84ae8b18fc12233ac6c81 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:17.999070 3545 raft_consensus.cc:729] T 44a17dadf4d84ae8b18fc12233ac6c81 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:13:17.999431 3545 consensus_queue.cc:260] T 44a17dadf4d84ae8b18fc12233ac6c81 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:18.001848 3545 ts_tablet_manager.cc:1398] T 44a17dadf4d84ae8b18fc12233ac6c81 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:13:18.012042 3529 ts_tablet_manager.cc:1875] T f5370f4730794e72806446e1e71d7c5f P f575ce92c0224eeb80b9fae5f6561fbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:18.012784 3528 ts_tablet_manager.cc:1888] T f5370f4730794e72806446e1e71d7c5f P 1f10f80f13c84e80b03eda7085a3b012: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:18.012902 3528 log.cc:1196] T f5370f4730794e72806446e1e71d7c5f P 1f10f80f13c84e80b03eda7085a3b012: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/wals/f5370f4730794e72806446e1e71d7c5f
I20240503 02:13:18.028019 546 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:18.048437 3533 tablet_bootstrap.cc:492] T 20019d63ee3e45418dc9c99bd25b72af P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:13:18.048671 3533 ts_tablet_manager.cc:1367] T 20019d63ee3e45418dc9c99bd25b72af P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.147s user 0.004s sys 0.000s
I20240503 02:13:18.049420 3533 raft_consensus.cc:348] T 20019d63ee3e45418dc9c99bd25b72af P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:18.050146 3533 raft_consensus.cc:374] T 20019d63ee3e45418dc9c99bd25b72af P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:18.050276 3533 raft_consensus.cc:729] T 20019d63ee3e45418dc9c99bd25b72af P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:13:18.050663 3533 consensus_queue.cc:260] T 20019d63ee3e45418dc9c99bd25b72af P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:18.051512 3533 ts_tablet_manager.cc:1398] T 20019d63ee3e45418dc9c99bd25b72af P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:13:18.051865 3533 tablet_bootstrap.cc:492] T 1758ec3783b24d3c820ed0d6dc118e69 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:13:18.054237 3533 tablet_bootstrap.cc:654] T 1758ec3783b24d3c820ed0d6dc118e69 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:18.063668 3528 ts_tablet_manager.cc:1909] T f5370f4730794e72806446e1e71d7c5f P 1f10f80f13c84e80b03eda7085a3b012: Deleting consensus metadata
I20240503 02:13:18.065332 29719 catalog_manager.cc:4830] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): tablet f5370f4730794e72806446e1e71d7c5f (table rebalancer_dynamic_table_new_name [id=d081257b79bd4fc9beade07061b17fba]) successfully deleted
I20240503 02:13:18.065554 3532 tablet_bootstrap.cc:492] T 1758ec3783b24d3c820ed0d6dc118e69 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:13:18.065866 3532 ts_tablet_manager.cc:1367] T 1758ec3783b24d3c820ed0d6dc118e69 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.114s user 0.002s sys 0.001s
I20240503 02:13:18.066612 3532 raft_consensus.cc:348] T 1758ec3783b24d3c820ed0d6dc118e69 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:18.066972 3532 raft_consensus.cc:374] T 1758ec3783b24d3c820ed0d6dc118e69 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:18.067077 3532 raft_consensus.cc:729] T 1758ec3783b24d3c820ed0d6dc118e69 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:13:18.067379 3532 consensus_queue.cc:260] T 1758ec3783b24d3c820ed0d6dc118e69 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:18.073940 3534 ts_tablet_manager.cc:1888] T f5370f4730794e72806446e1e71d7c5f P 9a4d92bc25374fb3960fde7d10be309d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:18.074954 3534 log.cc:1196] T f5370f4730794e72806446e1e71d7c5f P 9a4d92bc25374fb3960fde7d10be309d: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal/wals/f5370f4730794e72806446e1e71d7c5f
I20240503 02:13:18.078975 3541 tablet_bootstrap.cc:492] T 1758ec3783b24d3c820ed0d6dc118e69 P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:18.079173 3541 ts_tablet_manager.cc:1367] T 1758ec3783b24d3c820ed0d6dc118e69 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.156s user 0.038s sys 0.043s
I20240503 02:13:18.079968 3541 raft_consensus.cc:348] T 1758ec3783b24d3c820ed0d6dc118e69 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:18.080263 3541 raft_consensus.cc:374] T 1758ec3783b24d3c820ed0d6dc118e69 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:18.080384 3541 raft_consensus.cc:729] T 1758ec3783b24d3c820ed0d6dc118e69 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:18.080652 3541 consensus_queue.cc:260] T 1758ec3783b24d3c820ed0d6dc118e69 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:18.082612 3541 ts_tablet_manager.cc:1398] T 1758ec3783b24d3c820ed0d6dc118e69 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:13:18.082628 3532 ts_tablet_manager.cc:1398] T 1758ec3783b24d3c820ed0d6dc118e69 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.017s user 0.001s sys 0.000s
I20240503 02:13:18.083012 3541 tablet_bootstrap.cc:492] T 20019d63ee3e45418dc9c99bd25b72af P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:18.084791 3538 tablet_bootstrap.cc:492] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:18.085064 3541 tablet_bootstrap.cc:654] T 20019d63ee3e45418dc9c99bd25b72af P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:18.085140 3538 ts_tablet_manager.cc:1367] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.130s user 0.004s sys 0.002s
I20240503 02:13:18.086056 3538 raft_consensus.cc:348] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:18.086344 3538 raft_consensus.cc:374] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:18.086475 3538 raft_consensus.cc:729] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:18.086751 3538 consensus_queue.cc:260] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:18.088086 3533 tablet_bootstrap.cc:492] T 1758ec3783b24d3c820ed0d6dc118e69 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:13:18.088305 3533 ts_tablet_manager.cc:1367] T 1758ec3783b24d3c820ed0d6dc118e69 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.003s
I20240503 02:13:18.089083 3533 raft_consensus.cc:348] T 1758ec3783b24d3c820ed0d6dc118e69 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:18.089547 3533 raft_consensus.cc:374] T 1758ec3783b24d3c820ed0d6dc118e69 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:18.089962 3533 raft_consensus.cc:729] T 1758ec3783b24d3c820ed0d6dc118e69 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:13:18.078047 3536 tablet_bootstrap.cc:492] T eec091fa0ba947d68577d562e538ab56 P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:18.090600 3536 ts_tablet_manager.cc:1367] T eec091fa0ba947d68577d562e538ab56 P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.177s user 0.002s sys 0.003s
I20240503 02:13:18.090867 3538 ts_tablet_manager.cc:1398] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 02:13:18.091110 3534 ts_tablet_manager.cc:1909] T f5370f4730794e72806446e1e71d7c5f P 9a4d92bc25374fb3960fde7d10be309d: Deleting consensus metadata
I20240503 02:13:18.091498 3536 raft_consensus.cc:348] T eec091fa0ba947d68577d562e538ab56 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:18.091691 3536 raft_consensus.cc:374] T eec091fa0ba947d68577d562e538ab56 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:18.091781 3536 raft_consensus.cc:729] T eec091fa0ba947d68577d562e538ab56 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:18.092008 3536 consensus_queue.cc:260] T eec091fa0ba947d68577d562e538ab56 P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:18.092545 3536 ts_tablet_manager.cc:1398] T eec091fa0ba947d68577d562e538ab56 P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:13:18.090368 3533 consensus_queue.cc:260] T 1758ec3783b24d3c820ed0d6dc118e69 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:18.094228 3498 raft_consensus.cc:482] T 1758ec3783b24d3c820ed0d6dc118e69 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:18.094338 3498 raft_consensus.cc:504] T 1758ec3783b24d3c820ed0d6dc118e69 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:18.095219 3498 leader_election.cc:288] T 1758ec3783b24d3c820ed0d6dc118e69 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:18.095820 29719 catalog_manager.cc:4830] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): tablet f5370f4730794e72806446e1e71d7c5f (table rebalancer_dynamic_table_new_name [id=d081257b79bd4fc9beade07061b17fba]) successfully deleted
I20240503 02:13:18.095983 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1758ec3783b24d3c820ed0d6dc118e69" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:13:18.096269 437 raft_consensus.cc:2454] T 1758ec3783b24d3c820ed0d6dc118e69 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:13:18.096297 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1758ec3783b24d3c820ed0d6dc118e69" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:13:18.096499 30004 raft_consensus.cc:2454] T 1758ec3783b24d3c820ed0d6dc118e69 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:13:18.096838 29809 leader_election.cc:302] T 1758ec3783b24d3c820ed0d6dc118e69 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429; no voters:
I20240503 02:13:18.097141 3533 ts_tablet_manager.cc:1398] T 1758ec3783b24d3c820ed0d6dc118e69 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 02:13:18.097380 3498 raft_consensus.cc:2781] T 1758ec3783b24d3c820ed0d6dc118e69 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:18.097461 3498 raft_consensus.cc:482] T 1758ec3783b24d3c820ed0d6dc118e69 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:18.097519 3498 raft_consensus.cc:3037] T 1758ec3783b24d3c820ed0d6dc118e69 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:18.100082 3498 raft_consensus.cc:504] T 1758ec3783b24d3c820ed0d6dc118e69 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:18.101420 435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1758ec3783b24d3c820ed0d6dc118e69" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:13:18.101531 3541 tablet_bootstrap.cc:492] T 20019d63ee3e45418dc9c99bd25b72af P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:18.101612 435 raft_consensus.cc:3037] T 1758ec3783b24d3c820ed0d6dc118e69 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:18.101744 3541 ts_tablet_manager.cc:1367] T 20019d63ee3e45418dc9c99bd25b72af P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.001s
I20240503 02:13:18.102614 3541 raft_consensus.cc:348] T 20019d63ee3e45418dc9c99bd25b72af P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:18.102931 3541 raft_consensus.cc:374] T 20019d63ee3e45418dc9c99bd25b72af P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:18.103020 3541 raft_consensus.cc:729] T 20019d63ee3e45418dc9c99bd25b72af P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:18.103281 3541 consensus_queue.cc:260] T 20019d63ee3e45418dc9c99bd25b72af P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:18.104593 435 raft_consensus.cc:2454] T 1758ec3783b24d3c820ed0d6dc118e69 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:13:18.104959 3541 ts_tablet_manager.cc:1398] T 20019d63ee3e45418dc9c99bd25b72af P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:18.105250 29809 leader_election.cc:302] T 1758ec3783b24d3c820ed0d6dc118e69 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:18.105311 3541 tablet_bootstrap.cc:492] T eec091fa0ba947d68577d562e538ab56 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:18.105479 3498 leader_election.cc:288] T 1758ec3783b24d3c820ed0d6dc118e69 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Requested vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:18.105479 2755 raft_consensus.cc:2781] T 1758ec3783b24d3c820ed0d6dc118e69 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:18.105633 2755 raft_consensus.cc:686] T 1758ec3783b24d3c820ed0d6dc118e69 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Becoming Leader. State: Replica: 16440e365430405fb10366ab70e2b429, State: Running, Role: LEADER
I20240503 02:13:18.097393 3533 tablet_bootstrap.cc:492] T 44a17dadf4d84ae8b18fc12233ac6c81 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:13:18.105873 2755 consensus_queue.cc:237] T 1758ec3783b24d3c820ed0d6dc118e69 P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:18.105928 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1758ec3783b24d3c820ed0d6dc118e69" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:13:18.106128 30004 raft_consensus.cc:3037] T 1758ec3783b24d3c820ed0d6dc118e69 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:18.107060 3533 tablet_bootstrap.cc:654] T 44a17dadf4d84ae8b18fc12233ac6c81 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:18.108829 30004 raft_consensus.cc:2454] T 1758ec3783b24d3c820ed0d6dc118e69 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:13:18.108892 3539 tablet_bootstrap.cc:492] T 20019d63ee3e45418dc9c99bd25b72af P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:13:18.109095 3539 ts_tablet_manager.cc:1367] T 20019d63ee3e45418dc9c99bd25b72af P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.162s user 0.006s sys 0.000s
I20240503 02:13:18.109993 3539 raft_consensus.cc:348] T 20019d63ee3e45418dc9c99bd25b72af P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:18.110330 3539 raft_consensus.cc:374] T 20019d63ee3e45418dc9c99bd25b72af P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:18.110467 3539 raft_consensus.cc:729] T 20019d63ee3e45418dc9c99bd25b72af P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:13:18.110790 3539 consensus_queue.cc:260] T 20019d63ee3e45418dc9c99bd25b72af P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:18.111776 3539 ts_tablet_manager.cc:1398] T 20019d63ee3e45418dc9c99bd25b72af P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:18.112219 3539 tablet_bootstrap.cc:492] T 44a17dadf4d84ae8b18fc12233ac6c81 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:13:18.114645 3541 tablet_bootstrap.cc:654] T eec091fa0ba947d68577d562e538ab56 P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:18.115761 3539 tablet_bootstrap.cc:654] T 44a17dadf4d84ae8b18fc12233ac6c81 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:18.133309 29729 catalog_manager.cc:5484] T 1758ec3783b24d3c820ed0d6dc118e69 P 16440e365430405fb10366ab70e2b429 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16440e365430405fb10366ab70e2b429 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:18.140694 3529 ts_tablet_manager.cc:1888] T f5370f4730794e72806446e1e71d7c5f P f575ce92c0224eeb80b9fae5f6561fbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:18.140925 3529 log.cc:1196] T f5370f4730794e72806446e1e71d7c5f P f575ce92c0224eeb80b9fae5f6561fbe: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal/wals/f5370f4730794e72806446e1e71d7c5f
I20240503 02:13:18.141523 3529 ts_tablet_manager.cc:1909] T f5370f4730794e72806446e1e71d7c5f P f575ce92c0224eeb80b9fae5f6561fbe: Deleting consensus metadata
I20240503 02:13:18.142668 29718 catalog_manager.cc:4830] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): tablet f5370f4730794e72806446e1e71d7c5f (table rebalancer_dynamic_table_new_name [id=d081257b79bd4fc9beade07061b17fba]) successfully deleted
I20240503 02:13:18.147687 3533 tablet_bootstrap.cc:492] T 44a17dadf4d84ae8b18fc12233ac6c81 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:13:18.147908 3533 ts_tablet_manager.cc:1367] T 44a17dadf4d84ae8b18fc12233ac6c81 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.000s
I20240503 02:13:18.148633 3533 raft_consensus.cc:348] T 44a17dadf4d84ae8b18fc12233ac6c81 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:18.148908 3533 raft_consensus.cc:374] T 44a17dadf4d84ae8b18fc12233ac6c81 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:18.148993 3533 raft_consensus.cc:729] T 44a17dadf4d84ae8b18fc12233ac6c81 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:13:18.149291 3533 consensus_queue.cc:260] T 44a17dadf4d84ae8b18fc12233ac6c81 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:18.149983 3533 ts_tablet_manager.cc:1398] T 44a17dadf4d84ae8b18fc12233ac6c81 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:13:18.151355 2324 raft_consensus.cc:482] T 44a17dadf4d84ae8b18fc12233ac6c81 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:18.151500 2324 raft_consensus.cc:504] T 44a17dadf4d84ae8b18fc12233ac6c81 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:18.152390 2324 leader_election.cc:288] T 44a17dadf4d84ae8b18fc12233ac6c81 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:13:18.153121 29874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a17dadf4d84ae8b18fc12233ac6c81" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:13:18.153343 29874 raft_consensus.cc:2454] T 44a17dadf4d84ae8b18fc12233ac6c81 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:13:18.154009 30202 leader_election.cc:302] T 44a17dadf4d84ae8b18fc12233ac6c81 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:13:18.155282 30132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a17dadf4d84ae8b18fc12233ac6c81" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
W20240503 02:13:18.157599 30199 leader_election.cc:341] T 44a17dadf4d84ae8b18fc12233ac6c81 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:18.158123 3511 raft_consensus.cc:2781] T 44a17dadf4d84ae8b18fc12233ac6c81 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:18.158203 3511 raft_consensus.cc:482] T 44a17dadf4d84ae8b18fc12233ac6c81 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:18.158268 3511 raft_consensus.cc:3037] T 44a17dadf4d84ae8b18fc12233ac6c81 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:18.161042 3511 raft_consensus.cc:504] T 44a17dadf4d84ae8b18fc12233ac6c81 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:18.161612 3511 leader_election.cc:288] T 44a17dadf4d84ae8b18fc12233ac6c81 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Requested vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:13:18.162789 29874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a17dadf4d84ae8b18fc12233ac6c81" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:13:18.163012 29874 raft_consensus.cc:3037] T 44a17dadf4d84ae8b18fc12233ac6c81 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:18.167032 30132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a17dadf4d84ae8b18fc12233ac6c81" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
W20240503 02:13:18.167750 30199 leader_election.cc:341] T 44a17dadf4d84ae8b18fc12233ac6c81 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:18.169404 3541 tablet_bootstrap.cc:492] T eec091fa0ba947d68577d562e538ab56 P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:18.169622 3541 ts_tablet_manager.cc:1367] T eec091fa0ba947d68577d562e538ab56 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.064s user 0.002s sys 0.007s
I20240503 02:13:18.169621 29874 raft_consensus.cc:2454] T 44a17dadf4d84ae8b18fc12233ac6c81 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:13:18.170044 30202 leader_election.cc:302] T 44a17dadf4d84ae8b18fc12233ac6c81 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16440e365430405fb10366ab70e2b429, 1f10f80f13c84e80b03eda7085a3b012; no voters: a9094240860e43bfb129193f3bfd5be6
I20240503 02:13:18.170492 2324 raft_consensus.cc:2781] T 44a17dadf4d84ae8b18fc12233ac6c81 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:18.170625 2324 raft_consensus.cc:686] T 44a17dadf4d84ae8b18fc12233ac6c81 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Becoming Leader. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Running, Role: LEADER
I20240503 02:13:18.170526 3541 raft_consensus.cc:348] T eec091fa0ba947d68577d562e538ab56 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:18.170717 3541 raft_consensus.cc:374] T eec091fa0ba947d68577d562e538ab56 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:18.170794 3541 raft_consensus.cc:729] T eec091fa0ba947d68577d562e538ab56 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:18.170874 2324 consensus_queue.cc:237] T 44a17dadf4d84ae8b18fc12233ac6c81 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:18.171041 3541 consensus_queue.cc:260] T eec091fa0ba947d68577d562e538ab56 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:18.175806 3541 ts_tablet_manager.cc:1398] T eec091fa0ba947d68577d562e538ab56 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 02:13:18.180119 3539 tablet_bootstrap.cc:492] T 44a17dadf4d84ae8b18fc12233ac6c81 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:13:18.180351 3539 ts_tablet_manager.cc:1367] T 44a17dadf4d84ae8b18fc12233ac6c81 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.068s user 0.000s sys 0.003s
I20240503 02:13:18.183126 3539 raft_consensus.cc:348] T 44a17dadf4d84ae8b18fc12233ac6c81 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:18.183426 3539 raft_consensus.cc:374] T 44a17dadf4d84ae8b18fc12233ac6c81 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:18.183542 3539 raft_consensus.cc:729] T 44a17dadf4d84ae8b18fc12233ac6c81 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:13:18.183923 3539 consensus_queue.cc:260] T 44a17dadf4d84ae8b18fc12233ac6c81 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:18.184455 3539 ts_tablet_manager.cc:1398] T 44a17dadf4d84ae8b18fc12233ac6c81 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:13:18.194272 3271 raft_consensus.cc:482] T 44a17dadf4d84ae8b18fc12233ac6c81 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:18.194432 3271 raft_consensus.cc:504] T 44a17dadf4d84ae8b18fc12233ac6c81 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:18.200232 3524 raft_consensus.cc:482] T eec091fa0ba947d68577d562e538ab56 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:18.200413 3524 raft_consensus.cc:504] T eec091fa0ba947d68577d562e538ab56 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:18.201200 3524 leader_election.cc:288] T eec091fa0ba947d68577d562e538ab56 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:13:18.201906 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec091fa0ba947d68577d562e538ab56" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:18.202165 567 raft_consensus.cc:2454] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 0.
I20240503 02:13:18.202349 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec091fa0ba947d68577d562e538ab56" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:13:18.202593 437 raft_consensus.cc:2454] T eec091fa0ba947d68577d562e538ab56 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 0.
I20240503 02:13:18.202770 32573 leader_election.cc:302] T eec091fa0ba947d68577d562e538ab56 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:18.203128 3524 raft_consensus.cc:2781] T eec091fa0ba947d68577d562e538ab56 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:18.203244 3524 raft_consensus.cc:482] T eec091fa0ba947d68577d562e538ab56 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:18.203328 3524 raft_consensus.cc:3037] T eec091fa0ba947d68577d562e538ab56 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:18.208074 3271 leader_election.cc:288] T 44a17dadf4d84ae8b18fc12233ac6c81 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:13:18.208920 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a17dadf4d84ae8b18fc12233ac6c81" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:13:18.209746 3524 raft_consensus.cc:504] T eec091fa0ba947d68577d562e538ab56 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:18.209846 29874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a17dadf4d84ae8b18fc12233ac6c81" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:13:18.210176 29874 raft_consensus.cc:2379] T 44a17dadf4d84ae8b18fc12233ac6c81 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a9094240860e43bfb129193f3bfd5be6 in current term 1: Already voted for candidate 1f10f80f13c84e80b03eda7085a3b012 in this term.
I20240503 02:13:18.210543 3524 leader_election.cc:288] T eec091fa0ba947d68577d562e538ab56 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 election: Requested vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:13:18.210583 30072 leader_election.cc:302] T 44a17dadf4d84ae8b18fc12233ac6c81 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6; no voters: 16440e365430405fb10366ab70e2b429, 1f10f80f13c84e80b03eda7085a3b012
I20240503 02:13:18.210973 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec091fa0ba947d68577d562e538ab56" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:13:18.211071 3271 raft_consensus.cc:3037] T 44a17dadf4d84ae8b18fc12233ac6c81 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:18.211200 567 raft_consensus.cc:3037] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:18.213706 3271 raft_consensus.cc:2726] T 44a17dadf4d84ae8b18fc12233ac6c81 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:18.214123 567 raft_consensus.cc:2454] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 1.
I20240503 02:13:18.214512 32573 leader_election.cc:302] T eec091fa0ba947d68577d562e538ab56 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:18.217214 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec091fa0ba947d68577d562e538ab56" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:13:18.217425 437 raft_consensus.cc:3037] T eec091fa0ba947d68577d562e538ab56 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:18.220309 3524 raft_consensus.cc:2781] T eec091fa0ba947d68577d562e538ab56 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:18.220566 3524 raft_consensus.cc:686] T eec091fa0ba947d68577d562e538ab56 P 9a4d92bc25374fb3960fde7d10be309d [term 1 LEADER]: Becoming Leader. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Running, Role: LEADER
I20240503 02:13:18.220219 437 raft_consensus.cc:2454] T eec091fa0ba947d68577d562e538ab56 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 1.
I20240503 02:13:18.223492 3524 consensus_queue.cc:237] T eec091fa0ba947d68577d562e538ab56 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:18.229372 29729 catalog_manager.cc:5484] T 44a17dadf4d84ae8b18fc12233ac6c81 P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4). New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:18.263842 29731 catalog_manager.cc:5484] T eec091fa0ba947d68577d562e538ab56 P 9a4d92bc25374fb3960fde7d10be309d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5). New cstate: current_term: 1 leader_uuid: "9a4d92bc25374fb3960fde7d10be309d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:18.291926 2755 raft_consensus.cc:482] T 20019d63ee3e45418dc9c99bd25b72af P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:18.292107 2755 raft_consensus.cc:504] T 20019d63ee3e45418dc9c99bd25b72af P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:18.293068 2755 leader_election.cc:288] T 20019d63ee3e45418dc9c99bd25b72af P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:13:18.293721 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20019d63ee3e45418dc9c99bd25b72af" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:13:18.293977 437 raft_consensus.cc:2454] T 20019d63ee3e45418dc9c99bd25b72af P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:13:18.294346 29809 leader_election.cc:302] T 20019d63ee3e45418dc9c99bd25b72af P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:18.294351 30132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20019d63ee3e45418dc9c99bd25b72af" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:13:18.294616 30132 raft_consensus.cc:2454] T 20019d63ee3e45418dc9c99bd25b72af P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:13:18.295065 2755 raft_consensus.cc:2781] T 20019d63ee3e45418dc9c99bd25b72af P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:18.295171 2755 raft_consensus.cc:482] T 20019d63ee3e45418dc9c99bd25b72af P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:18.295256 2755 raft_consensus.cc:3037] T 20019d63ee3e45418dc9c99bd25b72af P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:18.298465 2755 raft_consensus.cc:504] T 20019d63ee3e45418dc9c99bd25b72af P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:18.299607 2755 leader_election.cc:288] T 20019d63ee3e45418dc9c99bd25b72af P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Requested vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:13:18.299602 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20019d63ee3e45418dc9c99bd25b72af" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:13:18.299635 30132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20019d63ee3e45418dc9c99bd25b72af" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:13:18.299808 30132 raft_consensus.cc:3037] T 20019d63ee3e45418dc9c99bd25b72af P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:18.299845 437 raft_consensus.cc:3037] T 20019d63ee3e45418dc9c99bd25b72af P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:18.302678 437 raft_consensus.cc:2454] T 20019d63ee3e45418dc9c99bd25b72af P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:13:18.302695 30132 raft_consensus.cc:2454] T 20019d63ee3e45418dc9c99bd25b72af P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:13:18.303136 29808 leader_election.cc:302] T 20019d63ee3e45418dc9c99bd25b72af P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:13:18.303512 2755 raft_consensus.cc:2781] T 20019d63ee3e45418dc9c99bd25b72af P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:18.303622 2755 raft_consensus.cc:686] T 20019d63ee3e45418dc9c99bd25b72af P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Becoming Leader. State: Replica: 16440e365430405fb10366ab70e2b429, State: Running, Role: LEADER
I20240503 02:13:18.303879 2755 consensus_queue.cc:237] T 20019d63ee3e45418dc9c99bd25b72af P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:18.309245 29731 catalog_manager.cc:5484] T 20019d63ee3e45418dc9c99bd25b72af P 16440e365430405fb10366ab70e2b429 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16440e365430405fb10366ab70e2b429 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:18.317574 29874 consensus_queue.cc:237] T df37d88053e2419fbe40d03023d0c756 P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:18.319653 30266 raft_consensus.cc:1261] T df37d88053e2419fbe40d03023d0c756 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:18.319650 437 raft_consensus.cc:1261] T df37d88053e2419fbe40d03023d0c756 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:18.320328 3498 consensus_queue.cc:1035] T df37d88053e2419fbe40d03023d0c756 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:18.320607 2755 consensus_queue.cc:1035] T df37d88053e2419fbe40d03023d0c756 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:18.323523 2755 raft_consensus.cc:2932] T df37d88053e2419fbe40d03023d0c756 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:18.324258 30266 raft_consensus.cc:2932] T df37d88053e2419fbe40d03023d0c756 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:18.324769 437 raft_consensus.cc:2932] T df37d88053e2419fbe40d03023d0c756 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:18.332162 29731 catalog_manager.cc:5484] T df37d88053e2419fbe40d03023d0c756 P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:18.339978 30116 tablet_service.cc:1514] Processing DeleteTablet for tablet df37d88053e2419fbe40d03023d0c756 with delete_type TABLET_DATA_TOMBSTONED (TS a9094240860e43bfb129193f3bfd5be6 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44812
I20240503 02:13:18.340821 3584 tablet_replica.cc:330] T df37d88053e2419fbe40d03023d0c756 P a9094240860e43bfb129193f3bfd5be6: stopping tablet replica
I20240503 02:13:18.341174 3584 raft_consensus.cc:2229] T df37d88053e2419fbe40d03023d0c756 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:18.341634 3584 raft_consensus.cc:2258] T df37d88053e2419fbe40d03023d0c756 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:18.342429 3584 ts_tablet_manager.cc:1875] T df37d88053e2419fbe40d03023d0c756 P a9094240860e43bfb129193f3bfd5be6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:18.349162 3584 ts_tablet_manager.cc:1888] T df37d88053e2419fbe40d03023d0c756 P a9094240860e43bfb129193f3bfd5be6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:13:18.349328 3584 log.cc:1196] T df37d88053e2419fbe40d03023d0c756 P a9094240860e43bfb129193f3bfd5be6: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/wals/df37d88053e2419fbe40d03023d0c756
I20240503 02:13:18.350155 29718 catalog_manager.cc:4830] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): tablet df37d88053e2419fbe40d03023d0c756 (table rebalance_test_table_9 [id=e80de094521a4d71bc2fb5f3343275b8]) successfully deleted
I20240503 02:13:18.378769 29730 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:13:18.394882 30132 raft_consensus.cc:1261] T 44a17dadf4d84ae8b18fc12233ac6c81 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:18.396345 3511 consensus_queue.cc:1035] T 44a17dadf4d84ae8b18fc12233ac6c81 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:18.400835 29874 raft_consensus.cc:1261] T 44a17dadf4d84ae8b18fc12233ac6c81 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:18.401738 3511 consensus_queue.cc:1035] T 44a17dadf4d84ae8b18fc12233ac6c81 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:18.416119 29730 catalog_manager.cc:5576] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719) does not have the latest schema for tablet 44a17dadf4d84ae8b18fc12233ac6c81 (table rebalancer_dynamic_table [id=5a4aa82bed34461680c15babb4e47e3b]). Expected version 1 got 0
I20240503 02:13:18.417330 3597 tablet.cc:1722] T 44a17dadf4d84ae8b18fc12233ac6c81 P 1f10f80f13c84e80b03eda7085a3b012: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:18.418047 3597 tablet.cc:1620] T 44a17dadf4d84ae8b18fc12233ac6c81 P 1f10f80f13c84e80b03eda7085a3b012: MemRowSet was empty: no flush needed.
I20240503 02:13:18.422174 3598 tablet.cc:1722] T 44a17dadf4d84ae8b18fc12233ac6c81 P 16440e365430405fb10366ab70e2b429: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:18.422833 29729 catalog_manager.cc:5576] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679) does not have the latest schema for tablet 44a17dadf4d84ae8b18fc12233ac6c81 (table rebalancer_dynamic_table [id=5a4aa82bed34461680c15babb4e47e3b]). Expected version 1 got 0
I20240503 02:13:18.423592 3598 tablet.cc:1620] T 44a17dadf4d84ae8b18fc12233ac6c81 P 16440e365430405fb10366ab70e2b429: MemRowSet was empty: no flush needed.
I20240503 02:13:18.432610 29729 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=5a4aa82bed34461680c15babb4e47e3b] alter complete (version 1)
I20240503 02:13:18.446308 3600 tablet.cc:1722] T 44a17dadf4d84ae8b18fc12233ac6c81 P a9094240860e43bfb129193f3bfd5be6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:18.447026 3600 tablet.cc:1620] T 44a17dadf4d84ae8b18fc12233ac6c81 P a9094240860e43bfb129193f3bfd5be6: MemRowSet was empty: no flush needed.
I20240503 02:13:18.452723 29729 catalog_manager.cc:5576] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679) does not have the latest schema for tablet 44a17dadf4d84ae8b18fc12233ac6c81 (table rebalancer_dynamic_table [id=5a4aa82bed34461680c15babb4e47e3b]). Expected version 1 got 0
I20240503 02:13:18.546031 29731 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:13:18.549249 3498 consensus_queue.cc:1035] T 1758ec3783b24d3c820ed0d6dc118e69 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:18.560405 3597 tablet.cc:1722] T 44a17dadf4d84ae8b18fc12233ac6c81 P 1f10f80f13c84e80b03eda7085a3b012: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:18.560954 3597 tablet.cc:1620] T 44a17dadf4d84ae8b18fc12233ac6c81 P 1f10f80f13c84e80b03eda7085a3b012: MemRowSet was empty: no flush needed.
I20240503 02:13:18.562175 3600 tablet.cc:1722] T 44a17dadf4d84ae8b18fc12233ac6c81 P a9094240860e43bfb129193f3bfd5be6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:18.562624 3600 tablet.cc:1620] T 44a17dadf4d84ae8b18fc12233ac6c81 P a9094240860e43bfb129193f3bfd5be6: MemRowSet was empty: no flush needed.
I20240503 02:13:18.564164 3598 tablet.cc:1722] T 44a17dadf4d84ae8b18fc12233ac6c81 P 16440e365430405fb10366ab70e2b429: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:18.564580 3598 tablet.cc:1620] T 44a17dadf4d84ae8b18fc12233ac6c81 P 16440e365430405fb10366ab70e2b429: MemRowSet was empty: no flush needed.
I20240503 02:13:18.576002 29731 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=5a4aa82bed34461680c15babb4e47e3b] alter complete (version 2)
I20240503 02:13:18.598554 2947 consensus_queue.cc:1035] T 1758ec3783b24d3c820ed0d6dc118e69 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:18.681699 29725 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:13:18.688161 29986 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:13:18.692603 3597 tablet.cc:1722] T 44a17dadf4d84ae8b18fc12233ac6c81 P 1f10f80f13c84e80b03eda7085a3b012: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:18.695628 3598 tablet.cc:1722] T 44a17dadf4d84ae8b18fc12233ac6c81 P 16440e365430405fb10366ab70e2b429: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:18.703760 29853 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:18.715263 3600 tablet.cc:1722] T 44a17dadf4d84ae8b18fc12233ac6c81 P a9094240860e43bfb129193f3bfd5be6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:18.719530 29725 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table_new_name [id=5a4aa82bed34461680c15babb4e47e3b] alter complete (version 3)
I20240503 02:13:18.720782 32754 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:18.740305 3524 consensus_queue.cc:1035] T eec091fa0ba947d68577d562e538ab56 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:18.765518 30245 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:13:18.807158 30116 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:13:18.823089 3652 consensus_queue.cc:1035] T eec091fa0ba947d68577d562e538ab56 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:18.823436 29725 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:18.823644 29725 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:18.828136 29725 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet 44a17dadf4d84ae8b18fc12233ac6c81
I20240503 02:13:18.829387 30116 tablet_service.cc:1514] Processing DeleteTablet for tablet 44a17dadf4d84ae8b18fc12233ac6c81 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:18 UTC) from {username='slave'} at 127.0.0.1:44812
I20240503 02:13:18.829622 3584 tablet_replica.cc:330] T 44a17dadf4d84ae8b18fc12233ac6c81 P a9094240860e43bfb129193f3bfd5be6: stopping tablet replica
I20240503 02:13:18.829696 30245 tablet_service.cc:1514] Processing DeleteTablet for tablet 44a17dadf4d84ae8b18fc12233ac6c81 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:18 UTC) from {username='slave'} at 127.0.0.1:49856
I20240503 02:13:18.829802 3584 raft_consensus.cc:2229] T 44a17dadf4d84ae8b18fc12233ac6c81 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:18.830005 3584 raft_consensus.cc:2258] T 44a17dadf4d84ae8b18fc12233ac6c81 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:18.830091 29853 tablet_service.cc:1514] Processing DeleteTablet for tablet 44a17dadf4d84ae8b18fc12233ac6c81 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:18 UTC) from {username='slave'} at 127.0.0.1:58744
I20240503 02:13:18.830987 3584 ts_tablet_manager.cc:1875] T 44a17dadf4d84ae8b18fc12233ac6c81 P a9094240860e43bfb129193f3bfd5be6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:18.837157 3658 tablet_replica.cc:330] T 44a17dadf4d84ae8b18fc12233ac6c81 P 1f10f80f13c84e80b03eda7085a3b012: stopping tablet replica
I20240503 02:13:18.837437 3658 raft_consensus.cc:2229] T 44a17dadf4d84ae8b18fc12233ac6c81 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:18.838758 3584 ts_tablet_manager.cc:1888] T 44a17dadf4d84ae8b18fc12233ac6c81 P a9094240860e43bfb129193f3bfd5be6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:18.838868 3584 log.cc:1196] T 44a17dadf4d84ae8b18fc12233ac6c81 P a9094240860e43bfb129193f3bfd5be6: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/wals/44a17dadf4d84ae8b18fc12233ac6c81
I20240503 02:13:18.839255 3584 ts_tablet_manager.cc:1909] T 44a17dadf4d84ae8b18fc12233ac6c81 P a9094240860e43bfb129193f3bfd5be6: Deleting consensus metadata
I20240503 02:13:18.841570 29718 catalog_manager.cc:4830] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): tablet 44a17dadf4d84ae8b18fc12233ac6c81 (table rebalancer_dynamic_table_new_name [id=5a4aa82bed34461680c15babb4e47e3b]) successfully deleted
I20240503 02:13:18.847374 416 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:18.847975 3658 raft_consensus.cc:2258] T 44a17dadf4d84ae8b18fc12233ac6c81 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:18.850423 3659 tablet_replica.cc:330] T 44a17dadf4d84ae8b18fc12233ac6c81 P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:13:18.849179 3658 ts_tablet_manager.cc:1875] T 44a17dadf4d84ae8b18fc12233ac6c81 P 1f10f80f13c84e80b03eda7085a3b012: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:18.852586 3659 raft_consensus.cc:2229] T 44a17dadf4d84ae8b18fc12233ac6c81 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:18.852897 3659 raft_consensus.cc:2258] T 44a17dadf4d84ae8b18fc12233ac6c81 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:18.856657 2947 consensus_queue.cc:1035] T 20019d63ee3e45418dc9c99bd25b72af P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:18.868263 546 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:18.880023 29725 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_extra_table_44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:18.887990 32621 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:13:18.894522 2947 consensus_queue.cc:1035] T 20019d63ee3e45418dc9c99bd25b72af P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:18.898308 30116 tablet_service.cc:1467] Processing CreateTablet for tablet 5c46d27e89ae412a9f18c27f18b68347 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=d7c477fdad9640c89f1806779ac9630f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:18.898648 30245 tablet_service.cc:1467] Processing CreateTablet for tablet 5c46d27e89ae412a9f18c27f18b68347 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=d7c477fdad9640c89f1806779ac9630f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:18.898780 30116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c46d27e89ae412a9f18c27f18b68347. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:18.899034 30245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c46d27e89ae412a9f18c27f18b68347. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:18.900281 32754 tablet_service.cc:1467] Processing CreateTablet for tablet 3351f8900c7446179da1fbe65f835e0a (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=d7c477fdad9640c89f1806779ac9630f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:18.900774 32754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3351f8900c7446179da1fbe65f835e0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:18.902240 32621 tablet_service.cc:1467] Processing CreateTablet for tablet 3351f8900c7446179da1fbe65f835e0a (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=d7c477fdad9640c89f1806779ac9630f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:18.903084 32621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3351f8900c7446179da1fbe65f835e0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:18.904758 546 tablet_service.cc:1467] Processing CreateTablet for tablet 5c46d27e89ae412a9f18c27f18b68347 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=d7c477fdad9640c89f1806779ac9630f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:18.905292 546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c46d27e89ae412a9f18c27f18b68347. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:18.905671 32619 tablet_service.cc:1467] Processing CreateTablet for tablet 20e81486fbbe44e8888fb7c5bb5b355e (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=d7c477fdad9640c89f1806779ac9630f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:18.906118 32619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20e81486fbbe44e8888fb7c5bb5b355e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:18.910946 32755 tablet_service.cc:1467] Processing CreateTablet for tablet 20e81486fbbe44e8888fb7c5bb5b355e (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=d7c477fdad9640c89f1806779ac9630f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:18.911360 32755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20e81486fbbe44e8888fb7c5bb5b355e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:18.915192 29986 tablet_service.cc:1467] Processing CreateTablet for tablet 20e81486fbbe44e8888fb7c5bb5b355e (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=d7c477fdad9640c89f1806779ac9630f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:18.933192 416 tablet_service.cc:1467] Processing CreateTablet for tablet 3351f8900c7446179da1fbe65f835e0a (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=d7c477fdad9640c89f1806779ac9630f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:18.933336 29733 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:13:18.933657 416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3351f8900c7446179da1fbe65f835e0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:18.937350 29986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20e81486fbbe44e8888fb7c5bb5b355e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:18.938313 3658 ts_tablet_manager.cc:1888] T 44a17dadf4d84ae8b18fc12233ac6c81 P 1f10f80f13c84e80b03eda7085a3b012: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:18.942039 3658 log.cc:1196] T 44a17dadf4d84ae8b18fc12233ac6c81 P 1f10f80f13c84e80b03eda7085a3b012: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/wals/44a17dadf4d84ae8b18fc12233ac6c81
I20240503 02:13:18.942548 3658 ts_tablet_manager.cc:1909] T 44a17dadf4d84ae8b18fc12233ac6c81 P 1f10f80f13c84e80b03eda7085a3b012: Deleting consensus metadata
I20240503 02:13:18.944571 29719 catalog_manager.cc:4830] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): tablet 44a17dadf4d84ae8b18fc12233ac6c81 (table rebalancer_dynamic_table_new_name [id=5a4aa82bed34461680c15babb4e47e3b]) successfully deleted
I20240503 02:13:18.952970 3659 ts_tablet_manager.cc:1875] T 44a17dadf4d84ae8b18fc12233ac6c81 P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:18.957293 3665 tablet_bootstrap.cc:492] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:13:18.960062 547 tablet_service.cc:1467] Processing CreateTablet for tablet b710a07ee26843d7bd0a9f05c9ccb0c8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=da15d536905a40f480d70996c71db7bb]), partition=
I20240503 02:13:18.960498 547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b710a07ee26843d7bd0a9f05c9ccb0c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:18.961704 3662 tablet_bootstrap.cc:492] T 5c46d27e89ae412a9f18c27f18b68347 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:13:18.963196 3662 tablet_bootstrap.cc:654] T 5c46d27e89ae412a9f18c27f18b68347 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:18.966760 3663 tablet_bootstrap.cc:492] T 20e81486fbbe44e8888fb7c5bb5b355e P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:18.969005 3663 tablet_bootstrap.cc:654] T 20e81486fbbe44e8888fb7c5bb5b355e P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:18.969720 3666 tablet_bootstrap.cc:492] T 5c46d27e89ae412a9f18c27f18b68347 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:18.971892 3666 tablet_bootstrap.cc:654] T 5c46d27e89ae412a9f18c27f18b68347 P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:18.973356 29853 tablet_service.cc:1467] Processing CreateTablet for tablet b710a07ee26843d7bd0a9f05c9ccb0c8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=da15d536905a40f480d70996c71db7bb]), partition=
I20240503 02:13:18.973805 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b710a07ee26843d7bd0a9f05c9ccb0c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:18.977834 3667 tablet_bootstrap.cc:492] T 3351f8900c7446179da1fbe65f835e0a P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:18.980458 3665 tablet_bootstrap.cc:654] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:18.986284 3667 tablet_bootstrap.cc:654] T 3351f8900c7446179da1fbe65f835e0a P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:18.971637 32621 tablet_service.cc:1467] Processing CreateTablet for tablet b710a07ee26843d7bd0a9f05c9ccb0c8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=da15d536905a40f480d70996c71db7bb]), partition=
I20240503 02:13:18.988572 32621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b710a07ee26843d7bd0a9f05c9ccb0c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:18.989018 3668 tablet_bootstrap.cc:492] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:13:18.990912 3665 tablet_bootstrap.cc:492] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:13:18.991108 3665 ts_tablet_manager.cc:1367] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.004s
I20240503 02:13:18.992038 3665 raft_consensus.cc:348] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:18.992415 3665 raft_consensus.cc:374] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:18.992551 3665 raft_consensus.cc:729] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:13:18.994446 3665 consensus_queue.cc:260] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:18.995169 3665 ts_tablet_manager.cc:1398] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:13:18.996701 3664 tablet_bootstrap.cc:492] T 20e81486fbbe44e8888fb7c5bb5b355e P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:18.997262 3662 tablet_bootstrap.cc:492] T 5c46d27e89ae412a9f18c27f18b68347 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:13:18.997459 3662 ts_tablet_manager.cc:1367] T 5c46d27e89ae412a9f18c27f18b68347 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.001s
I20240503 02:13:18.998265 3662 raft_consensus.cc:348] T 5c46d27e89ae412a9f18c27f18b68347 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:18.998507 3662 raft_consensus.cc:374] T 5c46d27e89ae412a9f18c27f18b68347 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:18.998611 3662 raft_consensus.cc:729] T 5c46d27e89ae412a9f18c27f18b68347 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:13:19.001170 3662 consensus_queue.cc:260] T 5c46d27e89ae412a9f18c27f18b68347 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:19.004382 3662 ts_tablet_manager.cc:1398] T 5c46d27e89ae412a9f18c27f18b68347 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20240503 02:13:19.020193 3668 tablet_bootstrap.cc:654] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:19.026693 3664 tablet_bootstrap.cc:654] T 20e81486fbbe44e8888fb7c5bb5b355e P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:19.030227 3667 tablet_bootstrap.cc:492] T 3351f8900c7446179da1fbe65f835e0a P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:19.030483 3667 ts_tablet_manager.cc:1367] T 3351f8900c7446179da1fbe65f835e0a P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.053s user 0.004s sys 0.001s
I20240503 02:13:19.035692 3674 tablet_bootstrap.cc:492] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:13:19.036222 3667 raft_consensus.cc:348] T 3351f8900c7446179da1fbe65f835e0a P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:19.036550 3667 raft_consensus.cc:374] T 3351f8900c7446179da1fbe65f835e0a P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:19.036753 3667 raft_consensus.cc:729] T 3351f8900c7446179da1fbe65f835e0a P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:19.037360 3599 raft_consensus.cc:482] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:19.037525 3599 raft_consensus.cc:504] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:19.038235 3599 leader_election.cc:288] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:13:19.038830 30135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c46d27e89ae412a9f18c27f18b68347" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:13:19.039038 30135 raft_consensus.cc:2454] T 5c46d27e89ae412a9f18c27f18b68347 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:13:19.039031 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c46d27e89ae412a9f18c27f18b68347" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:19.039433 30199 leader_election.cc:302] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:13:19.037549 3667 consensus_queue.cc:260] T 3351f8900c7446179da1fbe65f835e0a P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
W20240503 02:13:19.039762 30199 leader_election.cc:341] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:19.040063 3599 raft_consensus.cc:2781] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:19.040143 3599 raft_consensus.cc:482] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:19.040210 3599 raft_consensus.cc:3037] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:19.040488 3667 ts_tablet_manager.cc:1398] T 3351f8900c7446179da1fbe65f835e0a P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.010s user 0.001s sys 0.002s
I20240503 02:13:19.041317 3663 tablet_bootstrap.cc:492] T 20e81486fbbe44e8888fb7c5bb5b355e P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:19.041576 3663 ts_tablet_manager.cc:1367] T 20e81486fbbe44e8888fb7c5bb5b355e P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.075s user 0.001s sys 0.003s
I20240503 02:13:19.042752 3674 tablet_bootstrap.cc:654] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:19.042694 3663 raft_consensus.cc:348] T 20e81486fbbe44e8888fb7c5bb5b355e P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:19.043036 3663 raft_consensus.cc:374] T 20e81486fbbe44e8888fb7c5bb5b355e P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:19.043231 3663 raft_consensus.cc:729] T 20e81486fbbe44e8888fb7c5bb5b355e P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:19.044768 3599 raft_consensus.cc:504] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:19.045956 30135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c46d27e89ae412a9f18c27f18b68347" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:13:19.046145 30135 raft_consensus.cc:3037] T 5c46d27e89ae412a9f18c27f18b68347 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:19.048774 30135 raft_consensus.cc:2454] T 5c46d27e89ae412a9f18c27f18b68347 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:13:19.044673 3659 ts_tablet_manager.cc:1888] T 44a17dadf4d84ae8b18fc12233ac6c81 P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:19.049254 30199 leader_election.cc:302] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:13:19.049603 2324 raft_consensus.cc:2781] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:19.049816 3659 log.cc:1196] T 44a17dadf4d84ae8b18fc12233ac6c81 P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/44a17dadf4d84ae8b18fc12233ac6c81
I20240503 02:13:19.050762 3666 tablet_bootstrap.cc:492] T 5c46d27e89ae412a9f18c27f18b68347 P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:19.050997 3666 ts_tablet_manager.cc:1367] T 5c46d27e89ae412a9f18c27f18b68347 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.081s user 0.002s sys 0.003s
I20240503 02:13:19.051064 3659 ts_tablet_manager.cc:1909] T 44a17dadf4d84ae8b18fc12233ac6c81 P 16440e365430405fb10366ab70e2b429: Deleting consensus metadata
I20240503 02:13:19.052605 3666 raft_consensus.cc:348] T 5c46d27e89ae412a9f18c27f18b68347 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:19.053201 3666 raft_consensus.cc:374] T 5c46d27e89ae412a9f18c27f18b68347 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:19.054107 3666 raft_consensus.cc:729] T 5c46d27e89ae412a9f18c27f18b68347 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:19.053982 3663 consensus_queue.cc:260] T 20e81486fbbe44e8888fb7c5bb5b355e P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:19.054478 3666 consensus_queue.cc:260] T 5c46d27e89ae412a9f18c27f18b68347 P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:19.054872 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c46d27e89ae412a9f18c27f18b68347" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:13:19.055162 567 raft_consensus.cc:3037] T 5c46d27e89ae412a9f18c27f18b68347 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:19.055720 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet 44a17dadf4d84ae8b18fc12233ac6c81 (table rebalancer_dynamic_table_new_name [id=5a4aa82bed34461680c15babb4e47e3b]) successfully deleted
I20240503 02:13:19.056653 3666 ts_tablet_manager.cc:1398] T 5c46d27e89ae412a9f18c27f18b68347 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 02:13:19.057061 3666 tablet_bootstrap.cc:492] T b710a07ee26843d7bd0a9f05c9ccb0c8 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:19.060806 567 raft_consensus.cc:2454] T 5c46d27e89ae412a9f18c27f18b68347 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:13:19.060851 3664 tablet_bootstrap.cc:492] T 20e81486fbbe44e8888fb7c5bb5b355e P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:19.061015 3664 ts_tablet_manager.cc:1367] T 20e81486fbbe44e8888fb7c5bb5b355e P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.065s user 0.004s sys 0.000s
I20240503 02:13:19.062021 3664 raft_consensus.cc:348] T 20e81486fbbe44e8888fb7c5bb5b355e P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:19.062304 3664 raft_consensus.cc:374] T 20e81486fbbe44e8888fb7c5bb5b355e P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:19.062487 3664 raft_consensus.cc:729] T 20e81486fbbe44e8888fb7c5bb5b355e P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:19.063514 3599 leader_election.cc:288] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Requested vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:13:19.064556 3664 consensus_queue.cc:260] T 20e81486fbbe44e8888fb7c5bb5b355e P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:19.064536 3674 tablet_bootstrap.cc:492] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:13:19.067920 3674 ts_tablet_manager.cc:1367] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
I20240503 02:13:19.068495 3664 ts_tablet_manager.cc:1398] T 20e81486fbbe44e8888fb7c5bb5b355e P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240503 02:13:19.069768 3664 tablet_bootstrap.cc:492] T 3351f8900c7446179da1fbe65f835e0a P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:19.071923 3668 tablet_bootstrap.cc:492] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:13:19.069315 3674 raft_consensus.cc:348] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:19.073577 3674 raft_consensus.cc:374] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:19.073849 3674 raft_consensus.cc:729] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:13:19.074437 3674 consensus_queue.cc:260] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:19.074580 3668 ts_tablet_manager.cc:1367] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.086s user 0.004s sys 0.005s
I20240503 02:13:19.075994 3668 raft_consensus.cc:348] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:19.076231 3668 raft_consensus.cc:374] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:19.076323 3668 raft_consensus.cc:729] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:13:19.076567 3668 consensus_queue.cc:260] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:19.080633 3664 tablet_bootstrap.cc:654] T 3351f8900c7446179da1fbe65f835e0a P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:19.081595 3674 ts_tablet_manager.cc:1398] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240503 02:13:19.085208 2324 raft_consensus.cc:686] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Becoming Leader. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Running, Role: LEADER
I20240503 02:13:19.085608 2324 consensus_queue.cc:237] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:19.086969 3668 ts_tablet_manager.cc:1398] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20240503 02:13:19.090831 3663 ts_tablet_manager.cc:1398] T 20e81486fbbe44e8888fb7c5bb5b355e P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.048s user 0.002s sys 0.000s
I20240503 02:13:19.091241 3663 tablet_bootstrap.cc:492] T 3351f8900c7446179da1fbe65f835e0a P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:19.094733 3666 tablet_bootstrap.cc:654] T b710a07ee26843d7bd0a9f05c9ccb0c8 P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:19.096998 3663 tablet_bootstrap.cc:654] T 3351f8900c7446179da1fbe65f835e0a P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:19.100127 3666 tablet_bootstrap.cc:492] T b710a07ee26843d7bd0a9f05c9ccb0c8 P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:19.100540 3666 ts_tablet_manager.cc:1367] T b710a07ee26843d7bd0a9f05c9ccb0c8 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.043s user 0.000s sys 0.004s
I20240503 02:13:19.103513 3666 raft_consensus.cc:348] T b710a07ee26843d7bd0a9f05c9ccb0c8 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:19.103912 3666 raft_consensus.cc:374] T b710a07ee26843d7bd0a9f05c9ccb0c8 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:19.104063 3666 raft_consensus.cc:729] T b710a07ee26843d7bd0a9f05c9ccb0c8 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:19.104434 3666 consensus_queue.cc:260] T b710a07ee26843d7bd0a9f05c9ccb0c8 P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:19.105232 3666 ts_tablet_manager.cc:1398] T b710a07ee26843d7bd0a9f05c9ccb0c8 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:13:19.114128 3664 tablet_bootstrap.cc:492] T 3351f8900c7446179da1fbe65f835e0a P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:19.114387 3664 ts_tablet_manager.cc:1367] T 3351f8900c7446179da1fbe65f835e0a P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.045s user 0.000s sys 0.005s
I20240503 02:13:19.115572 3664 raft_consensus.cc:348] T 3351f8900c7446179da1fbe65f835e0a P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:19.116027 3664 raft_consensus.cc:374] T 3351f8900c7446179da1fbe65f835e0a P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:19.116153 3664 raft_consensus.cc:729] T 3351f8900c7446179da1fbe65f835e0a P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:19.116050 29725 catalog_manager.cc:5484] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4). New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:19.116537 3664 consensus_queue.cc:260] T 3351f8900c7446179da1fbe65f835e0a P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:19.117314 3664 ts_tablet_manager.cc:1398] T 3351f8900c7446179da1fbe65f835e0a P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:13:19.125943 3663 tablet_bootstrap.cc:492] T 3351f8900c7446179da1fbe65f835e0a P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:19.126145 3663 ts_tablet_manager.cc:1367] T 3351f8900c7446179da1fbe65f835e0a P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.000s
I20240503 02:13:19.127032 3663 raft_consensus.cc:348] T 3351f8900c7446179da1fbe65f835e0a P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:19.127279 3663 raft_consensus.cc:374] T 3351f8900c7446179da1fbe65f835e0a P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:19.127369 3663 raft_consensus.cc:729] T 3351f8900c7446179da1fbe65f835e0a P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:19.127684 3663 consensus_queue.cc:260] T 3351f8900c7446179da1fbe65f835e0a P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:19.128305 3663 ts_tablet_manager.cc:1398] T 3351f8900c7446179da1fbe65f835e0a P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:19.131225 3663 tablet_bootstrap.cc:492] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:19.133224 3663 tablet_bootstrap.cc:654] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:19.151083 3663 tablet_bootstrap.cc:492] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:19.151361 3663 ts_tablet_manager.cc:1367] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.002s
I20240503 02:13:19.152345 3663 raft_consensus.cc:348] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:19.152645 3663 raft_consensus.cc:374] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:19.152877 3663 raft_consensus.cc:729] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:19.153187 3663 consensus_queue.cc:260] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:19.159286 3663 ts_tablet_manager.cc:1398] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240503 02:13:19.185263 3653 raft_consensus.cc:482] T 20e81486fbbe44e8888fb7c5bb5b355e P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:19.185492 3653 raft_consensus.cc:504] T 20e81486fbbe44e8888fb7c5bb5b355e P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:19.186846 3653 leader_election.cc:288] T 20e81486fbbe44e8888fb7c5bb5b355e P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:13:19.187215 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20e81486fbbe44e8888fb7c5bb5b355e" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:13:19.187407 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20e81486fbbe44e8888fb7c5bb5b355e" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:13:19.187487 305 raft_consensus.cc:2454] T 20e81486fbbe44e8888fb7c5bb5b355e P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 0.
I20240503 02:13:19.187701 30004 raft_consensus.cc:2454] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 0.
I20240503 02:13:19.187985 32574 leader_election.cc:302] T 20e81486fbbe44e8888fb7c5bb5b355e P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, 9a4d92bc25374fb3960fde7d10be309d; no voters:
I20240503 02:13:19.188302 3653 raft_consensus.cc:2781] T 20e81486fbbe44e8888fb7c5bb5b355e P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:19.188477 3653 raft_consensus.cc:482] T 20e81486fbbe44e8888fb7c5bb5b355e P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:19.188607 3653 raft_consensus.cc:3037] T 20e81486fbbe44e8888fb7c5bb5b355e P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:19.194142 3653 raft_consensus.cc:504] T 20e81486fbbe44e8888fb7c5bb5b355e P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:19.195425 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20e81486fbbe44e8888fb7c5bb5b355e" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:13:19.195657 305 raft_consensus.cc:3037] T 20e81486fbbe44e8888fb7c5bb5b355e P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:19.196017 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20e81486fbbe44e8888fb7c5bb5b355e" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:13:19.196306 30004 raft_consensus.cc:3037] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:19.198616 305 raft_consensus.cc:2454] T 20e81486fbbe44e8888fb7c5bb5b355e P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 1.
I20240503 02:13:19.199256 32574 leader_election.cc:302] T 20e81486fbbe44e8888fb7c5bb5b355e P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, 9a4d92bc25374fb3960fde7d10be309d; no voters:
I20240503 02:13:19.199684 3651 raft_consensus.cc:2781] T 20e81486fbbe44e8888fb7c5bb5b355e P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:19.199849 3651 raft_consensus.cc:686] T 20e81486fbbe44e8888fb7c5bb5b355e P 9a4d92bc25374fb3960fde7d10be309d [term 1 LEADER]: Becoming Leader. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Running, Role: LEADER
I20240503 02:13:19.200191 3651 consensus_queue.cc:237] T 20e81486fbbe44e8888fb7c5bb5b355e P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:19.207208 30004 raft_consensus.cc:2454] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 1.
I20240503 02:13:19.208511 3653 leader_election.cc:288] T 20e81486fbbe44e8888fb7c5bb5b355e P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 election: Requested vote from peers 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:13:19.212591 29733 catalog_manager.cc:5484] T 20e81486fbbe44e8888fb7c5bb5b355e P 9a4d92bc25374fb3960fde7d10be309d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5). New cstate: current_term: 1 leader_uuid: "9a4d92bc25374fb3960fde7d10be309d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:19.248672 30135 consensus_queue.cc:237] T b6c9fd3cbbd5453897aa072600db569d P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:19.250941 437 raft_consensus.cc:1261] T b6c9fd3cbbd5453897aa072600db569d P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:19.251078 29874 raft_consensus.cc:1261] T b6c9fd3cbbd5453897aa072600db569d P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:19.251541 3271 consensus_queue.cc:1035] T b6c9fd3cbbd5453897aa072600db569d P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:19.251827 3433 consensus_queue.cc:1035] T b6c9fd3cbbd5453897aa072600db569d P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:19.266383 3180 raft_consensus.cc:482] T 3351f8900c7446179da1fbe65f835e0a P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:19.266644 3180 raft_consensus.cc:504] T 3351f8900c7446179da1fbe65f835e0a P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:19.267520 3180 leader_election.cc:288] T 3351f8900c7446179da1fbe65f835e0a P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:19.268440 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3351f8900c7446179da1fbe65f835e0a" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:13:19.268468 32639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3351f8900c7446179da1fbe65f835e0a" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:13:19.268734 305 raft_consensus.cc:2454] T 3351f8900c7446179da1fbe65f835e0a P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 0.
I20240503 02:13:19.268818 32639 raft_consensus.cc:2454] T 3351f8900c7446179da1fbe65f835e0a P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 0.
I20240503 02:13:19.269380 371 leader_election.cc:302] T 3351f8900c7446179da1fbe65f835e0a P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:19.269745 3180 raft_consensus.cc:2781] T 3351f8900c7446179da1fbe65f835e0a P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:19.269897 3180 raft_consensus.cc:482] T 3351f8900c7446179da1fbe65f835e0a P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:19.270005 3180 raft_consensus.cc:3037] T 3351f8900c7446179da1fbe65f835e0a P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:19.271307 3271 raft_consensus.cc:2932] T b6c9fd3cbbd5453897aa072600db569d P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:19.272464 29874 raft_consensus.cc:2932] T b6c9fd3cbbd5453897aa072600db569d P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:19.272845 3180 raft_consensus.cc:504] T 3351f8900c7446179da1fbe65f835e0a P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:19.273460 3180 leader_election.cc:288] T 3351f8900c7446179da1fbe65f835e0a P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Requested vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:19.274061 32639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3351f8900c7446179da1fbe65f835e0a" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d"
I20240503 02:13:19.274273 32639 raft_consensus.cc:3037] T 3351f8900c7446179da1fbe65f835e0a P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:19.275485 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3351f8900c7446179da1fbe65f835e0a" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:13:19.275703 305 raft_consensus.cc:3037] T 3351f8900c7446179da1fbe65f835e0a P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:19.278947 29733 catalog_manager.cc:5484] T b6c9fd3cbbd5453897aa072600db569d P 16440e365430405fb10366ab70e2b429 reported cstate change: config changed from index 4 to 5, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) evicted. New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:19.279685 32639 raft_consensus.cc:2454] T 3351f8900c7446179da1fbe65f835e0a P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 1.
I20240503 02:13:19.280172 371 leader_election.cc:302] T 3351f8900c7446179da1fbe65f835e0a P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:19.280484 3180 raft_consensus.cc:2781] T 3351f8900c7446179da1fbe65f835e0a P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:19.280647 3180 raft_consensus.cc:686] T 3351f8900c7446179da1fbe65f835e0a P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Becoming Leader. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Running, Role: LEADER
I20240503 02:13:19.280818 437 raft_consensus.cc:2932] T b6c9fd3cbbd5453897aa072600db569d P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:19.280979 3180 consensus_queue.cc:237] T 3351f8900c7446179da1fbe65f835e0a P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:19.283195 305 raft_consensus.cc:2454] T 3351f8900c7446179da1fbe65f835e0a P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 1.
I20240503 02:13:19.284060 29986 tablet_service.cc:1514] Processing DeleteTablet for tablet b6c9fd3cbbd5453897aa072600db569d with delete_type TABLET_DATA_TOMBSTONED (TS 155735c7f8b14997ac4c770aee2a9401 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:52158
I20240503 02:13:19.289494 29725 catalog_manager.cc:5484] T 3351f8900c7446179da1fbe65f835e0a P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7). New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:19.293121 3706 tablet_replica.cc:330] T b6c9fd3cbbd5453897aa072600db569d P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:13:19.293362 3706 raft_consensus.cc:2229] T b6c9fd3cbbd5453897aa072600db569d P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:19.293766 3706 raft_consensus.cc:2258] T b6c9fd3cbbd5453897aa072600db569d P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:19.294365 3706 ts_tablet_manager.cc:1875] T b6c9fd3cbbd5453897aa072600db569d P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:19.302999 3706 ts_tablet_manager.cc:1888] T b6c9fd3cbbd5453897aa072600db569d P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:13:19.303166 3706 log.cc:1196] T b6c9fd3cbbd5453897aa072600db569d P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/b6c9fd3cbbd5453897aa072600db569d
I20240503 02:13:19.304072 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet b6c9fd3cbbd5453897aa072600db569d (table rebalance_test_table_7 [id=f6554714a9d84d2d864dca86f3ad3642]) successfully deleted
I20240503 02:13:19.362735 2755 raft_consensus.cc:482] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:19.362942 2755 raft_consensus.cc:504] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:19.363705 2755 leader_election.cc:288] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:13:19.364166 566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b710a07ee26843d7bd0a9f05c9ccb0c8" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:19.364449 566 raft_consensus.cc:2454] T b710a07ee26843d7bd0a9f05c9ccb0c8 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:13:19.365300 29808 leader_election.cc:302] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:19.365417 32639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b710a07ee26843d7bd0a9f05c9ccb0c8" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:13:19.365715 32639 raft_consensus.cc:2454] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:13:19.367213 2755 raft_consensus.cc:2781] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:19.367327 2755 raft_consensus.cc:482] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:19.367413 2755 raft_consensus.cc:3037] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:19.370352 2755 raft_consensus.cc:504] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:19.372213 566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b710a07ee26843d7bd0a9f05c9ccb0c8" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:13:19.372457 566 raft_consensus.cc:3037] T b710a07ee26843d7bd0a9f05c9ccb0c8 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:19.374847 566 raft_consensus.cc:2454] T b710a07ee26843d7bd0a9f05c9ccb0c8 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:13:19.375376 29808 leader_election.cc:302] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:19.375684 2947 raft_consensus.cc:2781] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:19.375756 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b710a07ee26843d7bd0a9f05c9ccb0c8" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d"
I20240503 02:13:19.375876 2947 raft_consensus.cc:686] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Becoming Leader. State: Replica: 16440e365430405fb10366ab70e2b429, State: Running, Role: LEADER
I20240503 02:13:19.376001 32640 raft_consensus.cc:3037] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:19.376168 2947 consensus_queue.cc:237] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:19.380021 32640 raft_consensus.cc:2454] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:13:19.385816 2755 leader_election.cc:288] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Requested vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:13:19.389868 29725 catalog_manager.cc:5484] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 16440e365430405fb10366ab70e2b429 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16440e365430405fb10366ab70e2b429 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:19.542143 29733 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:13:19.579002 2324 consensus_queue.cc:1035] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:19.582079 566 raft_consensus.cc:1261] T b710a07ee26843d7bd0a9f05c9ccb0c8 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:19.583060 32640 raft_consensus.cc:1261] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:19.583390 2947 consensus_queue.cc:1035] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:19.583868 2947 consensus_queue.cc:1035] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:19.590750 3679 consensus_queue.cc:1035] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:19.606381 3651 consensus_queue.cc:1035] T 20e81486fbbe44e8888fb7c5bb5b355e P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:19.621150 3651 consensus_queue.cc:1035] T 20e81486fbbe44e8888fb7c5bb5b355e P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:19.640003 29729 catalog_manager.cc:5576] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233) does not have the latest schema for tablet b710a07ee26843d7bd0a9f05c9ccb0c8 (table rebalancer_dynamic_table [id=da15d536905a40f480d70996c71db7bb]). Expected version 1 got 0
I20240503 02:13:19.645721 29729 catalog_manager.cc:5576] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595) does not have the latest schema for tablet b710a07ee26843d7bd0a9f05c9ccb0c8 (table rebalancer_dynamic_table [id=da15d536905a40f480d70996c71db7bb]). Expected version 1 got 0
I20240503 02:13:19.657143 3760 tablet.cc:1722] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 16440e365430405fb10366ab70e2b429: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:19.657733 29729 catalog_manager.cc:5576] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595) does not have the latest schema for tablet b710a07ee26843d7bd0a9f05c9ccb0c8 (table rebalancer_dynamic_table [id=da15d536905a40f480d70996c71db7bb]). Expected version 1 got 0
I20240503 02:13:19.658113 3760 tablet.cc:1620] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 16440e365430405fb10366ab70e2b429: MemRowSet was empty: no flush needed.
I20240503 02:13:19.661373 3762 tablet.cc:1722] T b710a07ee26843d7bd0a9f05c9ccb0c8 P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:19.662228 3762 tablet.cc:1620] T b710a07ee26843d7bd0a9f05c9ccb0c8 P f575ce92c0224eeb80b9fae5f6561fbe: MemRowSet was empty: no flush needed.
I20240503 02:13:19.669189 29731 catalog_manager.cc:5576] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233) does not have the latest schema for tablet b710a07ee26843d7bd0a9f05c9ccb0c8 (table rebalancer_dynamic_table [id=da15d536905a40f480d70996c71db7bb]). Expected version 1 got 0
I20240503 02:13:19.653115 3761 tablet.cc:1722] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 9a4d92bc25374fb3960fde7d10be309d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:19.675129 3761 tablet.cc:1620] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 9a4d92bc25374fb3960fde7d10be309d: MemRowSet was empty: no flush needed.
I20240503 02:13:19.678120 29729 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=da15d536905a40f480d70996c71db7bb] alter complete (version 1)
I20240503 02:13:19.746558 30116 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:13:19.747352 3292 consensus_queue.cc:1035] T 3351f8900c7446179da1fbe65f835e0a P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:19.762351 29853 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:19.768756 30245 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:13:19.768934 29986 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:13:19.780012 546 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:19.789420 29725 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:13:19.803750 3760 tablet.cc:1722] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 16440e365430405fb10366ab70e2b429: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:19.804298 3760 tablet.cc:1620] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 16440e365430405fb10366ab70e2b429: MemRowSet was empty: no flush needed.
I20240503 02:13:19.807356 3761 tablet.cc:1722] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 9a4d92bc25374fb3960fde7d10be309d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:19.807845 3761 tablet.cc:1620] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 9a4d92bc25374fb3960fde7d10be309d: MemRowSet was empty: no flush needed.
I20240503 02:13:19.808732 3762 tablet.cc:1722] T b710a07ee26843d7bd0a9f05c9ccb0c8 P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:19.809249 3762 tablet.cc:1620] T b710a07ee26843d7bd0a9f05c9ccb0c8 P f575ce92c0224eeb80b9fae5f6561fbe: MemRowSet was empty: no flush needed.
I20240503 02:13:19.812286 29731 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_extra_table_46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:19.816813 32621 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:13:19.823208 29733 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=da15d536905a40f480d70996c71db7bb] alter complete (version 2)
I20240503 02:13:19.825243 32755 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:19.839784 29853 tablet_service.cc:1467] Processing CreateTablet for tablet f694fefb42d84318a6d91532105b3c90 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=95e60300d67c446987cfb1f3dbeecf3a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:19.840221 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f694fefb42d84318a6d91532105b3c90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:19.843019 29986 tablet_service.cc:1467] Processing CreateTablet for tablet 480f7b57c2c04c1399b945a023fbe41f (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=95e60300d67c446987cfb1f3dbeecf3a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:19.843554 29986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 480f7b57c2c04c1399b945a023fbe41f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:19.844969 30245 tablet_service.cc:1467] Processing CreateTablet for tablet 480f7b57c2c04c1399b945a023fbe41f (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=95e60300d67c446987cfb1f3dbeecf3a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:19.845259 30246 tablet_service.cc:1467] Processing CreateTablet for tablet e61f7581952e4d2ab0a473d2f965ba15 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=95e60300d67c446987cfb1f3dbeecf3a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:19.845412 30245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 480f7b57c2c04c1399b945a023fbe41f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:19.845429 29984 tablet_service.cc:1467] Processing CreateTablet for tablet e61f7581952e4d2ab0a473d2f965ba15 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=95e60300d67c446987cfb1f3dbeecf3a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:19.845625 30246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e61f7581952e4d2ab0a473d2f965ba15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:19.845880 29984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e61f7581952e4d2ab0a473d2f965ba15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:19.849480 30244 tablet_service.cc:1467] Processing CreateTablet for tablet f694fefb42d84318a6d91532105b3c90 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=95e60300d67c446987cfb1f3dbeecf3a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:19.849948 30244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f694fefb42d84318a6d91532105b3c90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:19.858244 32755 tablet_service.cc:1467] Processing CreateTablet for tablet 480f7b57c2c04c1399b945a023fbe41f (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=95e60300d67c446987cfb1f3dbeecf3a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:19.858395 32754 tablet_service.cc:1467] Processing CreateTablet for tablet f694fefb42d84318a6d91532105b3c90 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=95e60300d67c446987cfb1f3dbeecf3a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:19.858716 32755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 480f7b57c2c04c1399b945a023fbe41f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:19.859129 32754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f694fefb42d84318a6d91532105b3c90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:19.905208 3787 tablet_bootstrap.cc:492] T 480f7b57c2c04c1399b945a023fbe41f P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:13:19.906431 3788 tablet_bootstrap.cc:492] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:19.908445 3786 tablet_bootstrap.cc:492] T f694fefb42d84318a6d91532105b3c90 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:13:19.911940 3787 tablet_bootstrap.cc:654] T 480f7b57c2c04c1399b945a023fbe41f P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:19.921597 416 tablet_service.cc:1467] Processing CreateTablet for tablet e61f7581952e4d2ab0a473d2f965ba15 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=95e60300d67c446987cfb1f3dbeecf3a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:19.921921 3786 tablet_bootstrap.cc:654] T f694fefb42d84318a6d91532105b3c90 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:19.922073 416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e61f7581952e4d2ab0a473d2f965ba15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:19.928157 29731 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:13:19.940254 3762 tablet.cc:1722] T b710a07ee26843d7bd0a9f05c9ccb0c8 P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:19.940384 3761 tablet.cc:1722] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 9a4d92bc25374fb3960fde7d10be309d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:19.942232 3786 tablet_bootstrap.cc:492] T f694fefb42d84318a6d91532105b3c90 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:13:19.942376 3786 ts_tablet_manager.cc:1367] T f694fefb42d84318a6d91532105b3c90 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.000s
I20240503 02:13:19.943116 3786 raft_consensus.cc:348] T f694fefb42d84318a6d91532105b3c90 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:19.943315 3786 raft_consensus.cc:374] T f694fefb42d84318a6d91532105b3c90 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:19.943375 3786 raft_consensus.cc:729] T f694fefb42d84318a6d91532105b3c90 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:13:19.943603 3786 consensus_queue.cc:260] T f694fefb42d84318a6d91532105b3c90 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:19.944402 3786 ts_tablet_manager.cc:1398] T f694fefb42d84318a6d91532105b3c90 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:19.945428 3760 tablet.cc:1722] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 16440e365430405fb10366ab70e2b429: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:19.951488 29727 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table_new_name [id=da15d536905a40f480d70996c71db7bb] alter complete (version 3)
I20240503 02:13:19.985278 3785 consensus_queue.cc:1035] T 3351f8900c7446179da1fbe65f835e0a P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:20.005282 3788 tablet_bootstrap.cc:654] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:20.008708 3789 tablet_bootstrap.cc:492] T e61f7581952e4d2ab0a473d2f965ba15 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:13:20.010680 3789 tablet_bootstrap.cc:654] T e61f7581952e4d2ab0a473d2f965ba15 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:20.016012 3792 tablet_bootstrap.cc:492] T e61f7581952e4d2ab0a473d2f965ba15 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:20.021844 3792 tablet_bootstrap.cc:654] T e61f7581952e4d2ab0a473d2f965ba15 P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:20.031150 3788 tablet_bootstrap.cc:492] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:20.031400 3788 ts_tablet_manager.cc:1367] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.125s user 0.004s sys 0.000s
I20240503 02:13:20.032375 3788 raft_consensus.cc:348] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:20.032709 3788 raft_consensus.cc:374] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:20.032827 3788 raft_consensus.cc:729] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:20.033134 3788 consensus_queue.cc:260] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:20.034961 416 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:20.036751 3788 ts_tablet_manager.cc:1398] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
W20240503 02:13:20.037137 371 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.4:33935, user_credentials={real_user=slave}} blocked reactor thread for 52475.2us
I20240503 02:13:20.054586 3788 tablet_bootstrap.cc:492] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:20.057317 29727 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:20.057543 29727 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:20.058705 3789 tablet_bootstrap.cc:492] T e61f7581952e4d2ab0a473d2f965ba15 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:13:20.058928 3789 ts_tablet_manager.cc:1367] T e61f7581952e4d2ab0a473d2f965ba15 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.052s user 0.000s sys 0.004s
I20240503 02:13:20.062076 29727 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet b710a07ee26843d7bd0a9f05c9ccb0c8
I20240503 02:13:20.063046 546 tablet_service.cc:1514] Processing DeleteTablet for tablet b710a07ee26843d7bd0a9f05c9ccb0c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:20 UTC) from {username='slave'} at 127.0.0.1:39028
I20240503 02:13:20.063452 29853 tablet_service.cc:1514] Processing DeleteTablet for tablet b710a07ee26843d7bd0a9f05c9ccb0c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:20 UTC) from {username='slave'} at 127.0.0.1:58744
I20240503 02:13:20.063468 32621 tablet_service.cc:1514] Processing DeleteTablet for tablet b710a07ee26843d7bd0a9f05c9ccb0c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:20 UTC) from {username='slave'} at 127.0.0.1:38200
I20240503 02:13:20.072294 3789 raft_consensus.cc:348] T e61f7581952e4d2ab0a473d2f965ba15 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:20.075881 3789 raft_consensus.cc:374] T e61f7581952e4d2ab0a473d2f965ba15 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:20.078500 3789 raft_consensus.cc:729] T e61f7581952e4d2ab0a473d2f965ba15 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:13:20.078929 3789 consensus_queue.cc:260] T e61f7581952e4d2ab0a473d2f965ba15 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:20.081843 3789 ts_tablet_manager.cc:1398] T e61f7581952e4d2ab0a473d2f965ba15 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20240503 02:13:20.082609 3789 tablet_bootstrap.cc:492] T 480f7b57c2c04c1399b945a023fbe41f P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:13:20.085875 3801 tablet_replica.cc:330] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:13:20.086290 3789 tablet_bootstrap.cc:654] T 480f7b57c2c04c1399b945a023fbe41f P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:20.086583 3801 raft_consensus.cc:2229] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:20.087644 3801 raft_consensus.cc:2258] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.089712 3801 ts_tablet_manager.cc:1875] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:20.090060 3789 tablet_bootstrap.cc:492] T 480f7b57c2c04c1399b945a023fbe41f P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:13:20.090183 3789 ts_tablet_manager.cc:1367] T 480f7b57c2c04c1399b945a023fbe41f P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.002s
I20240503 02:13:20.091516 3789 raft_consensus.cc:348] T 480f7b57c2c04c1399b945a023fbe41f P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:20.091780 3789 raft_consensus.cc:374] T 480f7b57c2c04c1399b945a023fbe41f P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:20.091887 3789 raft_consensus.cc:729] T 480f7b57c2c04c1399b945a023fbe41f P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:13:20.092164 3789 consensus_queue.cc:260] T 480f7b57c2c04c1399b945a023fbe41f P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:20.093108 3789 ts_tablet_manager.cc:1398] T 480f7b57c2c04c1399b945a023fbe41f P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 02:13:20.098752 3801 ts_tablet_manager.cc:1888] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:20.098889 3801 log.cc:1196] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/b710a07ee26843d7bd0a9f05c9ccb0c8
I20240503 02:13:20.099314 3801 ts_tablet_manager.cc:1909] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 16440e365430405fb10366ab70e2b429: Deleting consensus metadata
I20240503 02:13:20.100505 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet b710a07ee26843d7bd0a9f05c9ccb0c8 (table rebalancer_dynamic_table_new_name [id=da15d536905a40f480d70996c71db7bb]) successfully deleted
I20240503 02:13:20.105587 3800 tablet_replica.cc:330] T b710a07ee26843d7bd0a9f05c9ccb0c8 P f575ce92c0224eeb80b9fae5f6561fbe: stopping tablet replica
I20240503 02:13:20.105861 3800 raft_consensus.cc:2229] T b710a07ee26843d7bd0a9f05c9ccb0c8 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:20.106094 3800 raft_consensus.cc:2258] T b710a07ee26843d7bd0a9f05c9ccb0c8 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.106642 3802 tablet_replica.cc:330] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 9a4d92bc25374fb3960fde7d10be309d: stopping tablet replica
I20240503 02:13:20.106858 3802 raft_consensus.cc:2229] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:20.107101 3802 raft_consensus.cc:2258] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.107995 3802 ts_tablet_manager.cc:1875] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 9a4d92bc25374fb3960fde7d10be309d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:20.113157 3800 ts_tablet_manager.cc:1875] T b710a07ee26843d7bd0a9f05c9ccb0c8 P f575ce92c0224eeb80b9fae5f6561fbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:20.115193 3792 tablet_bootstrap.cc:492] T e61f7581952e4d2ab0a473d2f965ba15 P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:20.115393 3792 ts_tablet_manager.cc:1367] T e61f7581952e4d2ab0a473d2f965ba15 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.102s user 0.005s sys 0.000s
I20240503 02:13:20.119989 3800 ts_tablet_manager.cc:1888] T b710a07ee26843d7bd0a9f05c9ccb0c8 P f575ce92c0224eeb80b9fae5f6561fbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:20.120160 3800 log.cc:1196] T b710a07ee26843d7bd0a9f05c9ccb0c8 P f575ce92c0224eeb80b9fae5f6561fbe: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal/wals/b710a07ee26843d7bd0a9f05c9ccb0c8
I20240503 02:13:20.120694 3800 ts_tablet_manager.cc:1909] T b710a07ee26843d7bd0a9f05c9ccb0c8 P f575ce92c0224eeb80b9fae5f6561fbe: Deleting consensus metadata
I20240503 02:13:20.121989 29718 catalog_manager.cc:4830] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): tablet b710a07ee26843d7bd0a9f05c9ccb0c8 (table rebalancer_dynamic_table_new_name [id=da15d536905a40f480d70996c71db7bb]) successfully deleted
I20240503 02:13:20.122377 3787 tablet_bootstrap.cc:492] T 480f7b57c2c04c1399b945a023fbe41f P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:13:20.122574 3787 ts_tablet_manager.cc:1367] T 480f7b57c2c04c1399b945a023fbe41f P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.222s user 0.003s sys 0.016s
I20240503 02:13:20.123323 3802 ts_tablet_manager.cc:1888] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 9a4d92bc25374fb3960fde7d10be309d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:20.123435 3802 log.cc:1196] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 9a4d92bc25374fb3960fde7d10be309d: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal/wals/b710a07ee26843d7bd0a9f05c9ccb0c8
I20240503 02:13:20.123951 3802 ts_tablet_manager.cc:1909] T b710a07ee26843d7bd0a9f05c9ccb0c8 P 9a4d92bc25374fb3960fde7d10be309d: Deleting consensus metadata
I20240503 02:13:20.124439 3787 raft_consensus.cc:348] T 480f7b57c2c04c1399b945a023fbe41f P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:20.124644 3787 raft_consensus.cc:374] T 480f7b57c2c04c1399b945a023fbe41f P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:20.124706 3787 raft_consensus.cc:729] T 480f7b57c2c04c1399b945a023fbe41f P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:13:20.125319 29719 catalog_manager.cc:4830] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): tablet b710a07ee26843d7bd0a9f05c9ccb0c8 (table rebalancer_dynamic_table_new_name [id=da15d536905a40f480d70996c71db7bb]) successfully deleted
I20240503 02:13:20.125928 3788 tablet_bootstrap.cc:654] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:20.124907 3787 consensus_queue.cc:260] T 480f7b57c2c04c1399b945a023fbe41f P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:20.127264 3787 ts_tablet_manager.cc:1398] T 480f7b57c2c04c1399b945a023fbe41f P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 02:13:20.133699 3787 tablet_bootstrap.cc:492] T e61f7581952e4d2ab0a473d2f965ba15 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:13:20.134275 3792 raft_consensus.cc:348] T e61f7581952e4d2ab0a473d2f965ba15 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:20.136355 3787 tablet_bootstrap.cc:654] T e61f7581952e4d2ab0a473d2f965ba15 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:20.138062 3792 raft_consensus.cc:374] T e61f7581952e4d2ab0a473d2f965ba15 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:20.140686 3792 raft_consensus.cc:729] T e61f7581952e4d2ab0a473d2f965ba15 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:20.143790 3792 consensus_queue.cc:260] T e61f7581952e4d2ab0a473d2f965ba15 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:20.144553 3792 ts_tablet_manager.cc:1398] T e61f7581952e4d2ab0a473d2f965ba15 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.029s user 0.003s sys 0.000s
I20240503 02:13:20.168530 29727 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:13:20.169653 3788 tablet_bootstrap.cc:492] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:20.169831 3788 ts_tablet_manager.cc:1367] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.115s user 0.000s sys 0.003s
I20240503 02:13:20.170598 3788 raft_consensus.cc:348] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:20.170818 3788 raft_consensus.cc:374] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:20.170886 3788 raft_consensus.cc:729] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:20.171101 3788 consensus_queue.cc:260] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:20.171633 3788 ts_tablet_manager.cc:1398] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:13:20.177202 3795 raft_consensus.cc:482] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:20.177383 3795 raft_consensus.cc:504] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:20.178452 3795 leader_election.cc:288] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:13:20.179857 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "480f7b57c2c04c1399b945a023fbe41f" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:13:20.180087 30005 raft_consensus.cc:2454] T 480f7b57c2c04c1399b945a023fbe41f P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:13:20.181624 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "480f7b57c2c04c1399b945a023fbe41f" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:13:20.181849 30265 raft_consensus.cc:2454] T 480f7b57c2c04c1399b945a023fbe41f P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:13:20.182353 32709 leader_election.cc:302] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 8a666e798fe8477d8dd1e51b385d5ead; no voters:
I20240503 02:13:20.182657 32621 tablet_service.cc:1467] Processing CreateTablet for tablet 19861ddccbe649c096d22fde936bf812 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b11f3b351d8b4d0980f4b390c3bb785f]), partition=
I20240503 02:13:20.183070 32621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19861ddccbe649c096d22fde936bf812. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:20.184219 546 tablet_service.cc:1467] Processing CreateTablet for tablet 19861ddccbe649c096d22fde936bf812 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b11f3b351d8b4d0980f4b390c3bb785f]), partition=
I20240503 02:13:20.184616 546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19861ddccbe649c096d22fde936bf812. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:20.184916 3795 raft_consensus.cc:2781] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:20.185057 3795 raft_consensus.cc:482] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:20.185166 3795 raft_consensus.cc:3037] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:20.186201 32754 tablet_service.cc:1467] Processing CreateTablet for tablet 19861ddccbe649c096d22fde936bf812 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b11f3b351d8b4d0980f4b390c3bb785f]), partition=
I20240503 02:13:20.187537 32754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19861ddccbe649c096d22fde936bf812. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:20.189348 3795 raft_consensus.cc:504] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:20.190390 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "480f7b57c2c04c1399b945a023fbe41f" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:13:20.190608 30005 raft_consensus.cc:3037] T 480f7b57c2c04c1399b945a023fbe41f P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:20.194150 30005 raft_consensus.cc:2454] T 480f7b57c2c04c1399b945a023fbe41f P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:13:20.195547 32709 leader_election.cc:302] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 8a666e798fe8477d8dd1e51b385d5ead; no voters:
I20240503 02:13:20.195649 3787 tablet_bootstrap.cc:492] T e61f7581952e4d2ab0a473d2f965ba15 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:13:20.195787 3787 ts_tablet_manager.cc:1367] T e61f7581952e4d2ab0a473d2f965ba15 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.062s user 0.012s sys 0.023s
I20240503 02:13:20.195863 3417 raft_consensus.cc:2781] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:20.195987 3417 raft_consensus.cc:686] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Becoming Leader. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Running, Role: LEADER
I20240503 02:13:20.196277 3417 consensus_queue.cc:237] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:20.196512 3787 raft_consensus.cc:348] T e61f7581952e4d2ab0a473d2f965ba15 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:20.196689 3787 raft_consensus.cc:374] T e61f7581952e4d2ab0a473d2f965ba15 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:20.196741 3787 raft_consensus.cc:729] T e61f7581952e4d2ab0a473d2f965ba15 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:13:20.196954 3787 consensus_queue.cc:260] T e61f7581952e4d2ab0a473d2f965ba15 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:20.198849 3787 ts_tablet_manager.cc:1398] T e61f7581952e4d2ab0a473d2f965ba15 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:20.199083 3787 tablet_bootstrap.cc:492] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:13:20.200675 3787 tablet_bootstrap.cc:654] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:20.205243 3788 tablet_bootstrap.cc:492] T 19861ddccbe649c096d22fde936bf812 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:20.206974 3788 tablet_bootstrap.cc:654] T 19861ddccbe649c096d22fde936bf812 P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:20.209180 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "480f7b57c2c04c1399b945a023fbe41f" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:13:20.209295 3795 leader_election.cc:288] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Requested vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:13:20.209395 30265 raft_consensus.cc:3037] T 480f7b57c2c04c1399b945a023fbe41f P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:20.212541 30265 raft_consensus.cc:2454] T 480f7b57c2c04c1399b945a023fbe41f P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:13:20.222952 3788 tablet_bootstrap.cc:492] T 19861ddccbe649c096d22fde936bf812 P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:20.223174 3788 ts_tablet_manager.cc:1367] T 19861ddccbe649c096d22fde936bf812 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.006s
I20240503 02:13:20.227383 29731 catalog_manager.cc:5484] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6). New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:20.228302 3788 raft_consensus.cc:348] T 19861ddccbe649c096d22fde936bf812 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:20.230305 3788 raft_consensus.cc:374] T 19861ddccbe649c096d22fde936bf812 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:20.230535 3788 raft_consensus.cc:729] T 19861ddccbe649c096d22fde936bf812 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:20.230887 3788 consensus_queue.cc:260] T 19861ddccbe649c096d22fde936bf812 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:20.232878 3788 ts_tablet_manager.cc:1398] T 19861ddccbe649c096d22fde936bf812 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:13:20.233407 3806 tablet_bootstrap.cc:492] T 19861ddccbe649c096d22fde936bf812 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:20.241597 3787 tablet_bootstrap.cc:492] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:13:20.241825 3787 ts_tablet_manager.cc:1367] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.043s user 0.009s sys 0.015s
I20240503 02:13:20.242796 3787 raft_consensus.cc:348] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:20.243044 3787 raft_consensus.cc:374] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:20.243134 3787 raft_consensus.cc:729] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:13:20.243362 3787 consensus_queue.cc:260] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:20.241312 3806 tablet_bootstrap.cc:654] T 19861ddccbe649c096d22fde936bf812 P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:20.246982 3809 tablet_bootstrap.cc:492] T 19861ddccbe649c096d22fde936bf812 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:20.248663 3809 tablet_bootstrap.cc:654] T 19861ddccbe649c096d22fde936bf812 P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:20.254760 3787 ts_tablet_manager.cc:1398] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20240503 02:13:20.273702 3809 tablet_bootstrap.cc:492] T 19861ddccbe649c096d22fde936bf812 P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:20.273923 3809 ts_tablet_manager.cc:1367] T 19861ddccbe649c096d22fde936bf812 P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.001s
I20240503 02:13:20.274641 3809 raft_consensus.cc:348] T 19861ddccbe649c096d22fde936bf812 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:20.277663 3809 raft_consensus.cc:374] T 19861ddccbe649c096d22fde936bf812 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:20.277944 3809 raft_consensus.cc:729] T 19861ddccbe649c096d22fde936bf812 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:20.278255 3809 consensus_queue.cc:260] T 19861ddccbe649c096d22fde936bf812 P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:20.278990 3809 ts_tablet_manager.cc:1398] T 19861ddccbe649c096d22fde936bf812 P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 02:13:20.282472 3806 tablet_bootstrap.cc:492] T 19861ddccbe649c096d22fde936bf812 P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:20.282677 3806 ts_tablet_manager.cc:1367] T 19861ddccbe649c096d22fde936bf812 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.049s user 0.011s sys 0.008s
I20240503 02:13:20.283731 3806 raft_consensus.cc:348] T 19861ddccbe649c096d22fde936bf812 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:20.284158 3806 raft_consensus.cc:374] T 19861ddccbe649c096d22fde936bf812 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:20.284320 3806 raft_consensus.cc:729] T 19861ddccbe649c096d22fde936bf812 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:20.284617 3806 consensus_queue.cc:260] T 19861ddccbe649c096d22fde936bf812 P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:20.285403 3806 ts_tablet_manager.cc:1398] T 19861ddccbe649c096d22fde936bf812 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240503 02:13:20.290175 3735 raft_consensus.cc:482] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:20.290328 3735 raft_consensus.cc:504] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:20.290995 3735 leader_election.cc:288] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:13:20.291702 29872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f694fefb42d84318a6d91532105b3c90" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:13:20.291970 29872 raft_consensus.cc:2454] T f694fefb42d84318a6d91532105b3c90 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:13:20.292260 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f694fefb42d84318a6d91532105b3c90" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:13:20.292460 30266 raft_consensus.cc:2454] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:13:20.292874 32711 leader_election.cc:302] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, 8a666e798fe8477d8dd1e51b385d5ead; no voters:
I20240503 02:13:20.293218 3735 raft_consensus.cc:2781] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:20.293324 3735 raft_consensus.cc:482] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:20.293397 3735 raft_consensus.cc:3037] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:20.297361 3735 raft_consensus.cc:504] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:20.299036 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f694fefb42d84318a6d91532105b3c90" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:13:20.299273 30266 raft_consensus.cc:3037] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:20.301328 3735 leader_election.cc:288] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Requested vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:13:20.301784 29872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f694fefb42d84318a6d91532105b3c90" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:13:20.301964 29872 raft_consensus.cc:3037] T f694fefb42d84318a6d91532105b3c90 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:20.302901 30266 raft_consensus.cc:2454] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:13:20.303934 29872 raft_consensus.cc:2454] T f694fefb42d84318a6d91532105b3c90 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:13:20.302155 3417 raft_consensus.cc:482] T 19861ddccbe649c096d22fde936bf812 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:20.304602 3417 raft_consensus.cc:504] T 19861ddccbe649c096d22fde936bf812 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:20.305169 3417 leader_election.cc:288] T 19861ddccbe649c096d22fde936bf812 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:13:20.303376 32709 leader_election.cc:302] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, 8a666e798fe8477d8dd1e51b385d5ead; no voters:
I20240503 02:13:20.305562 566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19861ddccbe649c096d22fde936bf812" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:20.305825 566 raft_consensus.cc:2454] T 19861ddccbe649c096d22fde936bf812 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:13:20.305938 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19861ddccbe649c096d22fde936bf812" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:13:20.306167 32640 raft_consensus.cc:2454] T 19861ddccbe649c096d22fde936bf812 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:13:20.306257 32708 leader_election.cc:302] T 19861ddccbe649c096d22fde936bf812 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:20.306509 3795 raft_consensus.cc:2781] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:20.307325 3417 raft_consensus.cc:2781] T 19861ddccbe649c096d22fde936bf812 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:20.307416 3417 raft_consensus.cc:482] T 19861ddccbe649c096d22fde936bf812 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:20.307526 3417 raft_consensus.cc:3037] T 19861ddccbe649c096d22fde936bf812 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:20.309468 3417 raft_consensus.cc:504] T 19861ddccbe649c096d22fde936bf812 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:20.310345 566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19861ddccbe649c096d22fde936bf812" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:13:20.310345 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19861ddccbe649c096d22fde936bf812" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d"
I20240503 02:13:20.310585 32640 raft_consensus.cc:3037] T 19861ddccbe649c096d22fde936bf812 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:20.310578 566 raft_consensus.cc:3037] T 19861ddccbe649c096d22fde936bf812 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:20.313287 32640 raft_consensus.cc:2454] T 19861ddccbe649c096d22fde936bf812 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:13:20.313694 32709 leader_election.cc:302] T 19861ddccbe649c096d22fde936bf812 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, 9a4d92bc25374fb3960fde7d10be309d; no voters:
I20240503 02:13:20.314005 3417 leader_election.cc:288] T 19861ddccbe649c096d22fde936bf812 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Requested vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:13:20.315455 3417 raft_consensus.cc:2781] T 19861ddccbe649c096d22fde936bf812 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:20.315623 3417 raft_consensus.cc:686] T 19861ddccbe649c096d22fde936bf812 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Becoming Leader. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Running, Role: LEADER
I20240503 02:13:20.315927 566 raft_consensus.cc:2454] T 19861ddccbe649c096d22fde936bf812 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:13:20.315920 3417 consensus_queue.cc:237] T 19861ddccbe649c096d22fde936bf812 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:20.323843 3795 raft_consensus.cc:686] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Becoming Leader. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Running, Role: LEADER
I20240503 02:13:20.324220 3795 consensus_queue.cc:237] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:20.329463 29731 catalog_manager.cc:5484] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6). New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:20.336088 29731 catalog_manager.cc:5484] T 19861ddccbe649c096d22fde936bf812 P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6). New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:20.360122 566 consensus_queue.cc:237] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:20.374819 29872 raft_consensus.cc:1261] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:20.375627 2946 consensus_queue.cc:1035] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:20.376157 32640 raft_consensus.cc:1261] T 6bd4fcc4faf44d148a03beaafc1c05ba P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:20.377628 3453 consensus_queue.cc:1035] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 02:13:20.388126 2946 raft_consensus.cc:2932] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:20.389636 32640 raft_consensus.cc:2932] T 6bd4fcc4faf44d148a03beaafc1c05ba P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:20.391975 3812 raft_consensus.cc:482] T e61f7581952e4d2ab0a473d2f965ba15 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:20.392102 3812 raft_consensus.cc:504] T e61f7581952e4d2ab0a473d2f965ba15 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:20.392818 3812 leader_election.cc:288] T e61f7581952e4d2ab0a473d2f965ba15 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:13:20.393733 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61f7581952e4d2ab0a473d2f965ba15" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:13:20.393942 30005 raft_consensus.cc:2454] T e61f7581952e4d2ab0a473d2f965ba15 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:13:20.394873 30200 leader_election.cc:302] T e61f7581952e4d2ab0a473d2f965ba15 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:13:20.394469 29872 raft_consensus.cc:2932] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:20.396298 433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61f7581952e4d2ab0a473d2f965ba15" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:13:20.396569 433 raft_consensus.cc:2454] T e61f7581952e4d2ab0a473d2f965ba15 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:13:20.397322 3811 raft_consensus.cc:2781] T e61f7581952e4d2ab0a473d2f965ba15 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:20.397409 3811 raft_consensus.cc:482] T e61f7581952e4d2ab0a473d2f965ba15 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:20.397480 3811 raft_consensus.cc:3037] T e61f7581952e4d2ab0a473d2f965ba15 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:20.400660 3811 raft_consensus.cc:504] T e61f7581952e4d2ab0a473d2f965ba15 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:20.402381 29727 catalog_manager.cc:5484] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: config changed from index 2 to 3, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New cstate: current_term: 1 leader_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:20.404965 3811 leader_election.cc:288] T e61f7581952e4d2ab0a473d2f965ba15 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Requested vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:13:20.405579 434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61f7581952e4d2ab0a473d2f965ba15" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:13:20.405823 434 raft_consensus.cc:3037] T e61f7581952e4d2ab0a473d2f965ba15 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:20.405843 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61f7581952e4d2ab0a473d2f965ba15" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:13:20.406052 30005 raft_consensus.cc:3037] T e61f7581952e4d2ab0a473d2f965ba15 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:20.408110 30005 raft_consensus.cc:2454] T e61f7581952e4d2ab0a473d2f965ba15 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:13:20.408442 434 raft_consensus.cc:2454] T e61f7581952e4d2ab0a473d2f965ba15 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:13:20.408481 30200 leader_election.cc:302] T e61f7581952e4d2ab0a473d2f965ba15 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:13:20.408895 3811 raft_consensus.cc:2781] T e61f7581952e4d2ab0a473d2f965ba15 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:20.409049 3811 raft_consensus.cc:686] T e61f7581952e4d2ab0a473d2f965ba15 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Becoming Leader. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Running, Role: LEADER
I20240503 02:13:20.409289 3811 consensus_queue.cc:237] T e61f7581952e4d2ab0a473d2f965ba15 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:20.413014 32754 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bd4fcc4faf44d148a03beaafc1c05ba with delete_type TABLET_DATA_TOMBSTONED (TS 8a666e798fe8477d8dd1e51b385d5ead not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46360
I20240503 02:13:20.413946 3841 tablet_replica.cc:330] T 6bd4fcc4faf44d148a03beaafc1c05ba P 8a666e798fe8477d8dd1e51b385d5ead: stopping tablet replica
I20240503 02:13:20.414252 3841 raft_consensus.cc:2229] T 6bd4fcc4faf44d148a03beaafc1c05ba P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:20.414686 3841 raft_consensus.cc:2258] T 6bd4fcc4faf44d148a03beaafc1c05ba P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.416143 3841 ts_tablet_manager.cc:1875] T 6bd4fcc4faf44d148a03beaafc1c05ba P 8a666e798fe8477d8dd1e51b385d5ead: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:20.421964 3841 ts_tablet_manager.cc:1888] T 6bd4fcc4faf44d148a03beaafc1c05ba P 8a666e798fe8477d8dd1e51b385d5ead: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:13:20.422087 3841 log.cc:1196] T 6bd4fcc4faf44d148a03beaafc1c05ba P 8a666e798fe8477d8dd1e51b385d5ead: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal/wals/6bd4fcc4faf44d148a03beaafc1c05ba
I20240503 02:13:20.422801 29719 catalog_manager.cc:4830] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): tablet 6bd4fcc4faf44d148a03beaafc1c05ba (table rebalancer_extra_table_10 [id=81455612cf1243e09bae574807b9303b]) successfully deleted
I20240503 02:13:20.430366 29729 catalog_manager.cc:5484] T e61f7581952e4d2ab0a473d2f965ba15 P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4). New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:20.440464 29729 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:13:20.454753 566 raft_consensus.cc:1261] T 19861ddccbe649c096d22fde936bf812 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:20.455123 32640 raft_consensus.cc:1261] T 19861ddccbe649c096d22fde936bf812 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:20.455780 3795 consensus_queue.cc:1035] T 19861ddccbe649c096d22fde936bf812 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:20.456236 3417 consensus_queue.cc:1035] T 19861ddccbe649c096d22fde936bf812 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:20.510496 29731 catalog_manager.cc:5576] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233) does not have the latest schema for tablet 19861ddccbe649c096d22fde936bf812 (table rebalancer_dynamic_table [id=b11f3b351d8b4d0980f4b390c3bb785f]). Expected version 1 got 0
I20240503 02:13:20.518370 3851 tablet.cc:1722] T 19861ddccbe649c096d22fde936bf812 P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:20.519106 3851 tablet.cc:1620] T 19861ddccbe649c096d22fde936bf812 P 8a666e798fe8477d8dd1e51b385d5ead: MemRowSet was empty: no flush needed.
I20240503 02:13:20.524497 3852 tablet.cc:1722] T 19861ddccbe649c096d22fde936bf812 P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:20.525199 3852 tablet.cc:1620] T 19861ddccbe649c096d22fde936bf812 P f575ce92c0224eeb80b9fae5f6561fbe: MemRowSet was empty: no flush needed.
I20240503 02:13:20.529456 29727 catalog_manager.cc:5576] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595) does not have the latest schema for tablet 19861ddccbe649c096d22fde936bf812 (table rebalancer_dynamic_table [id=b11f3b351d8b4d0980f4b390c3bb785f]). Expected version 1 got 0
I20240503 02:13:20.531737 29731 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=b11f3b351d8b4d0980f4b390c3bb785f] alter complete (version 1)
I20240503 02:13:20.536684 29727 catalog_manager.cc:5576] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233) does not have the latest schema for tablet 19861ddccbe649c096d22fde936bf812 (table rebalancer_dynamic_table [id=b11f3b351d8b4d0980f4b390c3bb785f]). Expected version 1 got 0
I20240503 02:13:20.540170 3853 tablet.cc:1722] T 19861ddccbe649c096d22fde936bf812 P 9a4d92bc25374fb3960fde7d10be309d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:20.541162 3853 tablet.cc:1620] T 19861ddccbe649c096d22fde936bf812 P 9a4d92bc25374fb3960fde7d10be309d: MemRowSet was empty: no flush needed.
I20240503 02:13:20.546244 29731 catalog_manager.cc:5576] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595) does not have the latest schema for tablet 19861ddccbe649c096d22fde936bf812 (table rebalancer_dynamic_table [id=b11f3b351d8b4d0980f4b390c3bb785f]). Expected version 1 got 0
I20240503 02:13:20.621687 3417 consensus_queue.cc:1035] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:20.629364 3417 consensus_queue.cc:1035] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:20.636503 29729 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:13:20.644789 3851 tablet.cc:1722] T 19861ddccbe649c096d22fde936bf812 P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:20.645329 3851 tablet.cc:1620] T 19861ddccbe649c096d22fde936bf812 P 8a666e798fe8477d8dd1e51b385d5ead: MemRowSet was empty: no flush needed.
I20240503 02:13:20.645388 3852 tablet.cc:1722] T 19861ddccbe649c096d22fde936bf812 P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:20.645867 3852 tablet.cc:1620] T 19861ddccbe649c096d22fde936bf812 P f575ce92c0224eeb80b9fae5f6561fbe: MemRowSet was empty: no flush needed.
I20240503 02:13:20.649169 3853 tablet.cc:1722] T 19861ddccbe649c096d22fde936bf812 P 9a4d92bc25374fb3960fde7d10be309d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:20.649590 3853 tablet.cc:1620] T 19861ddccbe649c096d22fde936bf812 P 9a4d92bc25374fb3960fde7d10be309d: MemRowSet was empty: no flush needed.
I20240503 02:13:20.661217 29727 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=b11f3b351d8b4d0980f4b390c3bb785f] alter complete (version 2)
I20240503 02:13:20.717598 3417 consensus_queue.cc:1035] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:20.729403 3417 consensus_queue.cc:1035] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:20.769928 29727 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:13:20.804570 3851 tablet.cc:1722] T 19861ddccbe649c096d22fde936bf812 P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:20.812260 3852 tablet.cc:1722] T 19861ddccbe649c096d22fde936bf812 P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:20.823513 3853 tablet.cc:1722] T 19861ddccbe649c096d22fde936bf812 P 9a4d92bc25374fb3960fde7d10be309d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:20.833717 29733 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table_new_name [id=b11f3b351d8b4d0980f4b390c3bb785f] alter complete (version 3)
I20240503 02:13:20.835037 3811 consensus_queue.cc:1035] T e61f7581952e4d2ab0a473d2f965ba15 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:20.845137 3811 consensus_queue.cc:1035] T e61f7581952e4d2ab0a473d2f965ba15 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:13:20.851387 30072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.1:36719, user_credentials={real_user=slave}} blocked reactor thread for 79282us
I20240503 02:13:20.940418 29733 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:20.940682 29733 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:20.946621 29733 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet 19861ddccbe649c096d22fde936bf812
I20240503 02:13:20.947782 546 tablet_service.cc:1514] Processing DeleteTablet for tablet 19861ddccbe649c096d22fde936bf812 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:20 UTC) from {username='slave'} at 127.0.0.1:39028
I20240503 02:13:20.947878 32754 tablet_service.cc:1514] Processing DeleteTablet for tablet 19861ddccbe649c096d22fde936bf812 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:20 UTC) from {username='slave'} at 127.0.0.1:46360
I20240503 02:13:20.948735 32621 tablet_service.cc:1514] Processing DeleteTablet for tablet 19861ddccbe649c096d22fde936bf812 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:20 UTC) from {username='slave'} at 127.0.0.1:38200
I20240503 02:13:20.949509 3923 tablet_replica.cc:330] T 19861ddccbe649c096d22fde936bf812 P 9a4d92bc25374fb3960fde7d10be309d: stopping tablet replica
I20240503 02:13:20.949838 3923 raft_consensus.cc:2229] T 19861ddccbe649c096d22fde936bf812 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:20.950107 3923 raft_consensus.cc:2258] T 19861ddccbe649c096d22fde936bf812 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.951392 3923 ts_tablet_manager.cc:1875] T 19861ddccbe649c096d22fde936bf812 P 9a4d92bc25374fb3960fde7d10be309d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:20.968839 3923 ts_tablet_manager.cc:1888] T 19861ddccbe649c096d22fde936bf812 P 9a4d92bc25374fb3960fde7d10be309d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:20.972438 3923 log.cc:1196] T 19861ddccbe649c096d22fde936bf812 P 9a4d92bc25374fb3960fde7d10be309d: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal/wals/19861ddccbe649c096d22fde936bf812
I20240503 02:13:20.973245 3923 ts_tablet_manager.cc:1909] T 19861ddccbe649c096d22fde936bf812 P 9a4d92bc25374fb3960fde7d10be309d: Deleting consensus metadata
I20240503 02:13:20.976804 29719 catalog_manager.cc:4830] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): tablet 19861ddccbe649c096d22fde936bf812 (table rebalancer_dynamic_table_new_name [id=b11f3b351d8b4d0980f4b390c3bb785f]) successfully deleted
I20240503 02:13:20.984707 32621 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:13:20.991953 29733 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_extra_table_48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:20.997669 29984 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:13:21.027359 30116 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:13:21.033136 32621 tablet_service.cc:1467] Processing CreateTablet for tablet 9b3b9d83e4c742bfa62624676cbb6a11 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=196b7559fe174e1b963b8d38f2240e48]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:21.033659 32621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b3b9d83e4c742bfa62624676cbb6a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:21.037803 30116 tablet_service.cc:1467] Processing CreateTablet for tablet 9b3b9d83e4c742bfa62624676cbb6a11 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=196b7559fe174e1b963b8d38f2240e48]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:21.038215 30116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b3b9d83e4c742bfa62624676cbb6a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:21.041694 29853 tablet_service.cc:1467] Processing CreateTablet for tablet 1b6cda34c02e4d24a126171c0063a7d6 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=196b7559fe174e1b963b8d38f2240e48]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:21.042119 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b6cda34c02e4d24a126171c0063a7d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:21.042212 30114 tablet_service.cc:1467] Processing CreateTablet for tablet 1b6cda34c02e4d24a126171c0063a7d6 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=196b7559fe174e1b963b8d38f2240e48]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:21.042600 30114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b6cda34c02e4d24a126171c0063a7d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:21.043996 30115 tablet_service.cc:1467] Processing CreateTablet for tablet 48b76d7755294d37860842a4a73ab28b (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=196b7559fe174e1b963b8d38f2240e48]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:21.044394 30115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48b76d7755294d37860842a4a73ab28b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:21.044893 547 tablet_service.cc:1467] Processing CreateTablet for tablet 48b76d7755294d37860842a4a73ab28b (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=196b7559fe174e1b963b8d38f2240e48]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:21.048141 547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48b76d7755294d37860842a4a73ab28b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:21.049558 29733 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:13:21.054525 416 tablet_service.cc:1467] Processing CreateTablet for tablet 9b3b9d83e4c742bfa62624676cbb6a11 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=196b7559fe174e1b963b8d38f2240e48]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:21.054589 417 tablet_service.cc:1467] Processing CreateTablet for tablet 48b76d7755294d37860842a4a73ab28b (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=196b7559fe174e1b963b8d38f2240e48]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:21.055179 32619 tablet_service.cc:1467] Processing CreateTablet for tablet 1b6cda34c02e4d24a126171c0063a7d6 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=196b7559fe174e1b963b8d38f2240e48]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:21.055606 32619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b6cda34c02e4d24a126171c0063a7d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:21.061276 416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b3b9d83e4c742bfa62624676cbb6a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:21.065632 417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48b76d7755294d37860842a4a73ab28b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:21.068727 30115 tablet_service.cc:1467] Processing CreateTablet for tablet 3c7789a4948a42a3bd528c629dab7aea (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f6d702a532b943d5922145591dbc9474]), partition=
I20240503 02:13:21.068975 415 tablet_service.cc:1467] Processing CreateTablet for tablet 3c7789a4948a42a3bd528c629dab7aea (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f6d702a532b943d5922145591dbc9474]), partition=
I20240503 02:13:21.069381 30115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c7789a4948a42a3bd528c629dab7aea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:21.069929 32755 tablet_service.cc:1467] Processing CreateTablet for tablet 3c7789a4948a42a3bd528c629dab7aea (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f6d702a532b943d5922145591dbc9474]), partition=
I20240503 02:13:21.070348 32755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c7789a4948a42a3bd528c629dab7aea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:21.070911 415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c7789a4948a42a3bd528c629dab7aea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:21.091377 3933 tablet_bootstrap.cc:492] T 3c7789a4948a42a3bd528c629dab7aea P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:21.094633 3935 tablet_bootstrap.cc:492] T 48b76d7755294d37860842a4a73ab28b P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:21.094627 3934 tablet_bootstrap.cc:492] T 3c7789a4948a42a3bd528c629dab7aea P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:21.096268 3934 tablet_bootstrap.cc:654] T 3c7789a4948a42a3bd528c629dab7aea P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:21.096863 3932 tablet_bootstrap.cc:492] T 1b6cda34c02e4d24a126171c0063a7d6 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:21.099366 3935 tablet_bootstrap.cc:654] T 48b76d7755294d37860842a4a73ab28b P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:21.099403 3933 tablet_bootstrap.cc:654] T 3c7789a4948a42a3bd528c629dab7aea P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:21.108791 3930 tablet_bootstrap.cc:492] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:13:21.110211 3932 tablet_bootstrap.cc:654] T 1b6cda34c02e4d24a126171c0063a7d6 P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:21.118273 3930 tablet_bootstrap.cc:654] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:21.126349 3933 tablet_bootstrap.cc:492] T 3c7789a4948a42a3bd528c629dab7aea P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:21.128428 3933 ts_tablet_manager.cc:1367] T 3c7789a4948a42a3bd528c629dab7aea P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.037s user 0.000s sys 0.005s
I20240503 02:13:21.128597 3934 tablet_bootstrap.cc:492] T 3c7789a4948a42a3bd528c629dab7aea P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:21.128878 3934 ts_tablet_manager.cc:1367] T 3c7789a4948a42a3bd528c629dab7aea P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.035s user 0.007s sys 0.005s
I20240503 02:13:21.129832 3935 tablet_bootstrap.cc:492] T 48b76d7755294d37860842a4a73ab28b P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:21.130089 3935 ts_tablet_manager.cc:1367] T 48b76d7755294d37860842a4a73ab28b P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.000s
I20240503 02:13:21.130545 3930 tablet_bootstrap.cc:492] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:13:21.130828 3930 ts_tablet_manager.cc:1367] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20240503 02:13:21.131608 3932 tablet_bootstrap.cc:492] T 1b6cda34c02e4d24a126171c0063a7d6 P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:21.131567 3930 raft_consensus.cc:348] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:21.131863 3930 raft_consensus.cc:374] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:21.131973 3930 raft_consensus.cc:729] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:13:21.132258 3930 consensus_queue.cc:260] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:21.132813 3932 ts_tablet_manager.cc:1367] T 1b6cda34c02e4d24a126171c0063a7d6 P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.002s
I20240503 02:13:21.133169 3935 raft_consensus.cc:348] T 48b76d7755294d37860842a4a73ab28b P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:21.133263 3933 raft_consensus.cc:348] T 3c7789a4948a42a3bd528c629dab7aea P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:21.133426 3935 raft_consensus.cc:374] T 48b76d7755294d37860842a4a73ab28b P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:21.133528 3935 raft_consensus.cc:729] T 48b76d7755294d37860842a4a73ab28b P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:21.133843 3935 consensus_queue.cc:260] T 48b76d7755294d37860842a4a73ab28b P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:21.134456 3930 ts_tablet_manager.cc:1398] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:13:21.134560 3934 raft_consensus.cc:348] T 3c7789a4948a42a3bd528c629dab7aea P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:21.134841 3934 raft_consensus.cc:374] T 3c7789a4948a42a3bd528c629dab7aea P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:21.134948 2946 raft_consensus.cc:482] T 48b76d7755294d37860842a4a73ab28b P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:21.135125 3934 raft_consensus.cc:729] T 3c7789a4948a42a3bd528c629dab7aea P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:21.135097 2946 raft_consensus.cc:504] T 48b76d7755294d37860842a4a73ab28b P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:21.135454 3930 tablet_bootstrap.cc:492] T 48b76d7755294d37860842a4a73ab28b P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:13:21.135748 2946 leader_election.cc:288] T 48b76d7755294d37860842a4a73ab28b P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:13:21.136240 30132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48b76d7755294d37860842a4a73ab28b" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:13:21.135700 3934 consensus_queue.cc:260] T 3c7789a4948a42a3bd528c629dab7aea P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
W20240503 02:13:21.136875 501 leader_election.cc:341] T 48b76d7755294d37860842a4a73ab28b P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:21.137744 3934 ts_tablet_manager.cc:1398] T 3c7789a4948a42a3bd528c629dab7aea P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 02:13:21.138541 3930 tablet_bootstrap.cc:654] T 48b76d7755294d37860842a4a73ab28b P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:21.148007 3931 tablet_bootstrap.cc:492] T 1b6cda34c02e4d24a126171c0063a7d6 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:13:21.149883 3931 tablet_bootstrap.cc:654] T 1b6cda34c02e4d24a126171c0063a7d6 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:21.151058 3922 tablet_replica.cc:330] T 19861ddccbe649c096d22fde936bf812 P 8a666e798fe8477d8dd1e51b385d5ead: stopping tablet replica
I20240503 02:13:21.151345 3922 raft_consensus.cc:2229] T 19861ddccbe649c096d22fde936bf812 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:21.156150 3935 ts_tablet_manager.cc:1398] T 48b76d7755294d37860842a4a73ab28b P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20240503 02:13:21.159081 30244 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:13:21.151777 3922 raft_consensus.cc:2258] T 19861ddccbe649c096d22fde936bf812 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:21.189924 3922 ts_tablet_manager.cc:1875] T 19861ddccbe649c096d22fde936bf812 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:21.191262 3930 tablet_bootstrap.cc:492] T 48b76d7755294d37860842a4a73ab28b P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:13:21.191401 3930 ts_tablet_manager.cc:1367] T 48b76d7755294d37860842a4a73ab28b P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.056s user 0.002s sys 0.021s
I20240503 02:13:21.192206 3930 raft_consensus.cc:348] T 48b76d7755294d37860842a4a73ab28b P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:21.192397 3930 raft_consensus.cc:374] T 48b76d7755294d37860842a4a73ab28b P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:21.192446 3930 raft_consensus.cc:729] T 48b76d7755294d37860842a4a73ab28b P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:13:21.192873 3921 tablet_replica.cc:330] T 19861ddccbe649c096d22fde936bf812 P f575ce92c0224eeb80b9fae5f6561fbe: stopping tablet replica
I20240503 02:13:21.192798 3930 consensus_queue.cc:260] T 48b76d7755294d37860842a4a73ab28b P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:21.193149 3921 raft_consensus.cc:2229] T 19861ddccbe649c096d22fde936bf812 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:21.193423 3921 raft_consensus.cc:2258] T 19861ddccbe649c096d22fde936bf812 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:21.196065 3930 ts_tablet_manager.cc:1398] T 48b76d7755294d37860842a4a73ab28b P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240503 02:13:21.196349 3930 tablet_bootstrap.cc:492] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:13:21.197461 3933 raft_consensus.cc:374] T 3c7789a4948a42a3bd528c629dab7aea P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:21.197655 3933 raft_consensus.cc:729] T 3c7789a4948a42a3bd528c629dab7aea P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:21.197995 3933 consensus_queue.cc:260] T 3c7789a4948a42a3bd528c629dab7aea P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:21.199028 3930 tablet_bootstrap.cc:654] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:21.200279 3933 ts_tablet_manager.cc:1398] T 3c7789a4948a42a3bd528c629dab7aea P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.071s user 0.000s sys 0.002s
I20240503 02:13:21.200696 3933 tablet_bootstrap.cc:492] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:21.205174 3932 raft_consensus.cc:348] T 1b6cda34c02e4d24a126171c0063a7d6 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:21.205482 3932 raft_consensus.cc:374] T 1b6cda34c02e4d24a126171c0063a7d6 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:21.205683 3932 raft_consensus.cc:729] T 1b6cda34c02e4d24a126171c0063a7d6 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:21.205977 3932 consensus_queue.cc:260] T 1b6cda34c02e4d24a126171c0063a7d6 P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:21.207980 3932 ts_tablet_manager.cc:1398] T 1b6cda34c02e4d24a126171c0063a7d6 P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.074s user 0.000s sys 0.002s
I20240503 02:13:21.208508 434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48b76d7755294d37860842a4a73ab28b" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:13:21.210119 3932 tablet_bootstrap.cc:492] T 9b3b9d83e4c742bfa62624676cbb6a11 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:21.225342 3933 tablet_bootstrap.cc:654] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:21.232201 32754 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:21.234902 3932 tablet_bootstrap.cc:654] T 9b3b9d83e4c742bfa62624676cbb6a11 P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:21.248457 3930 tablet_bootstrap.cc:492] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:13:21.249123 3930 ts_tablet_manager.cc:1367] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.053s user 0.004s sys 0.011s
I20240503 02:13:21.250609 3921 ts_tablet_manager.cc:1875] T 19861ddccbe649c096d22fde936bf812 P f575ce92c0224eeb80b9fae5f6561fbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:21.251082 3930 raft_consensus.cc:348] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:21.252207 3930 raft_consensus.cc:374] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:21.255630 3930 raft_consensus.cc:729] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:13:21.256098 3930 consensus_queue.cc:260] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:21.256667 3899 raft_consensus.cc:482] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:21.256799 3899 raft_consensus.cc:504] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:21.257398 3899 leader_election.cc:288] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:13:21.258529 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b6cda34c02e4d24a126171c0063a7d6" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:13:21.258760 32640 raft_consensus.cc:2454] T 1b6cda34c02e4d24a126171c0063a7d6 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 0.
I20240503 02:13:21.259240 30070 leader_election.cc:302] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:13:21.259574 3899 raft_consensus.cc:2781] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:21.259647 3899 raft_consensus.cc:482] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:21.259713 3899 raft_consensus.cc:3037] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:21.262455 3899 raft_consensus.cc:504] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:21.263484 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b6cda34c02e4d24a126171c0063a7d6" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d"
I20240503 02:13:21.263689 32640 raft_consensus.cc:3037] T 1b6cda34c02e4d24a126171c0063a7d6 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:21.266616 32640 raft_consensus.cc:2454] T 1b6cda34c02e4d24a126171c0063a7d6 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 1.
I20240503 02:13:21.267077 30070 leader_election.cc:302] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:13:21.267307 3900 raft_consensus.cc:2781] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:21.267390 3900 raft_consensus.cc:686] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Becoming Leader. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Running, Role: LEADER
I20240503 02:13:21.267657 3900 consensus_queue.cc:237] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:21.270172 3899 leader_election.cc:288] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Requested vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:13:21.275413 29872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b6cda34c02e4d24a126171c0063a7d6" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
W20240503 02:13:21.276157 30072 leader_election.cc:341] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:21.276572 29872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b6cda34c02e4d24a126171c0063a7d6" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
W20240503 02:13:21.277124 30072 leader_election.cc:341] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:21.283308 3930 ts_tablet_manager.cc:1398] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.033s user 0.001s sys 0.000s
I20240503 02:13:21.283660 3930 tablet_bootstrap.cc:492] T 3c7789a4948a42a3bd528c629dab7aea P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:13:21.286597 29733 catalog_manager.cc:5484] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9094240860e43bfb129193f3bfd5be6 (127.22.108.3). New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:21.291040 3795 raft_consensus.cc:482] T 3c7789a4948a42a3bd528c629dab7aea P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:21.291229 3795 raft_consensus.cc:504] T 3c7789a4948a42a3bd528c629dab7aea P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:21.292099 3795 leader_election.cc:288] T 3c7789a4948a42a3bd528c629dab7aea P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:13:21.293473 30132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c7789a4948a42a3bd528c629dab7aea" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:13:21.295518 415 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 02:13:21.297279 32708 leader_election.cc:341] T 3c7789a4948a42a3bd528c629dab7aea P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:21.293200 434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c7789a4948a42a3bd528c629dab7aea" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:13:21.299010 434 raft_consensus.cc:2454] T 3c7789a4948a42a3bd528c629dab7aea P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:13:21.299443 32709 leader_election.cc:302] T 3c7789a4948a42a3bd528c629dab7aea P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, fbbc2be6382e47908ecb7c68c367bd26; no voters: a9094240860e43bfb129193f3bfd5be6
I20240503 02:13:21.300071 3795 raft_consensus.cc:2781] T 3c7789a4948a42a3bd528c629dab7aea P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:21.300204 3795 raft_consensus.cc:482] T 3c7789a4948a42a3bd528c629dab7aea P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:21.300302 3795 raft_consensus.cc:3037] T 3c7789a4948a42a3bd528c629dab7aea P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:13:21.303454 502 leader_election.cc:341] T 48b76d7755294d37860842a4a73ab28b P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:21.303603 502 leader_election.cc:302] T 48b76d7755294d37860842a4a73ab28b P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f575ce92c0224eeb80b9fae5f6561fbe; no voters: a9094240860e43bfb129193f3bfd5be6, fbbc2be6382e47908ecb7c68c367bd26
I20240503 02:13:21.303910 3944 raft_consensus.cc:2726] T 48b76d7755294d37860842a4a73ab28b P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:21.308467 3795 raft_consensus.cc:504] T 3c7789a4948a42a3bd528c629dab7aea P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:21.310097 433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c7789a4948a42a3bd528c629dab7aea" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:13:21.310411 433 raft_consensus.cc:3037] T 3c7789a4948a42a3bd528c629dab7aea P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:21.313599 3930 tablet_bootstrap.cc:654] T 3c7789a4948a42a3bd528c629dab7aea P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:21.314138 30132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c7789a4948a42a3bd528c629dab7aea" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
W20240503 02:13:21.314808 32708 leader_election.cc:341] T 3c7789a4948a42a3bd528c629dab7aea P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:21.315204 3795 leader_election.cc:288] T 3c7789a4948a42a3bd528c629dab7aea P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Requested vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:13:21.321275 29853 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:21.322688 433 raft_consensus.cc:2454] T 3c7789a4948a42a3bd528c629dab7aea P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:13:21.323457 32709 leader_election.cc:302] T 3c7789a4948a42a3bd528c629dab7aea P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, fbbc2be6382e47908ecb7c68c367bd26; no voters: a9094240860e43bfb129193f3bfd5be6
I20240503 02:13:21.324028 3795 raft_consensus.cc:2781] T 3c7789a4948a42a3bd528c629dab7aea P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:21.325718 3795 raft_consensus.cc:686] T 3c7789a4948a42a3bd528c629dab7aea P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Becoming Leader. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Running, Role: LEADER
I20240503 02:13:21.326073 3795 consensus_queue.cc:237] T 3c7789a4948a42a3bd528c629dab7aea P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:21.329115 3931 tablet_bootstrap.cc:492] T 1b6cda34c02e4d24a126171c0063a7d6 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:13:21.329355 3931 ts_tablet_manager.cc:1367] T 1b6cda34c02e4d24a126171c0063a7d6 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.181s user 0.005s sys 0.000s
I20240503 02:13:21.332676 3931 raft_consensus.cc:348] T 1b6cda34c02e4d24a126171c0063a7d6 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:21.333904 3931 raft_consensus.cc:374] T 1b6cda34c02e4d24a126171c0063a7d6 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:21.334237 3931 raft_consensus.cc:729] T 1b6cda34c02e4d24a126171c0063a7d6 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:13:21.334605 3931 consensus_queue.cc:260] T 1b6cda34c02e4d24a126171c0063a7d6 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:21.338505 3931 ts_tablet_manager.cc:1398] T 1b6cda34c02e4d24a126171c0063a7d6 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240503 02:13:21.338598 3932 tablet_bootstrap.cc:492] T 9b3b9d83e4c742bfa62624676cbb6a11 P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:21.342017 3932 ts_tablet_manager.cc:1367] T 9b3b9d83e4c742bfa62624676cbb6a11 P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.131s user 0.003s sys 0.002s
I20240503 02:13:21.348670 3922 ts_tablet_manager.cc:1888] T 19861ddccbe649c096d22fde936bf812 P 8a666e798fe8477d8dd1e51b385d5ead: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:21.349155 3922 log.cc:1196] T 19861ddccbe649c096d22fde936bf812 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal/wals/19861ddccbe649c096d22fde936bf812
I20240503 02:13:21.350111 3922 ts_tablet_manager.cc:1909] T 19861ddccbe649c096d22fde936bf812 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting consensus metadata
I20240503 02:13:21.352308 29719 catalog_manager.cc:4830] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): tablet 19861ddccbe649c096d22fde936bf812 (table rebalancer_dynamic_table_new_name [id=b11f3b351d8b4d0980f4b390c3bb785f]) successfully deleted
I20240503 02:13:21.352351 29733 catalog_manager.cc:5484] T 3c7789a4948a42a3bd528c629dab7aea P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6). New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:21.347295 3932 raft_consensus.cc:348] T 9b3b9d83e4c742bfa62624676cbb6a11 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:21.356832 3932 raft_consensus.cc:374] T 9b3b9d83e4c742bfa62624676cbb6a11 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:21.356933 3932 raft_consensus.cc:729] T 9b3b9d83e4c742bfa62624676cbb6a11 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:21.357234 3932 consensus_queue.cc:260] T 9b3b9d83e4c742bfa62624676cbb6a11 P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:21.362502 3932 ts_tablet_manager.cc:1398] T 9b3b9d83e4c742bfa62624676cbb6a11 P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.020s user 0.000s sys 0.002s
I20240503 02:13:21.373317 546 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240503 02:13:21.373864 32573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.8:42233, user_credentials={real_user=slave}} blocked reactor thread for 45112.7us
I20240503 02:13:21.378518 3921 ts_tablet_manager.cc:1888] T 19861ddccbe649c096d22fde936bf812 P f575ce92c0224eeb80b9fae5f6561fbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:21.378670 3921 log.cc:1196] T 19861ddccbe649c096d22fde936bf812 P f575ce92c0224eeb80b9fae5f6561fbe: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal/wals/19861ddccbe649c096d22fde936bf812
I20240503 02:13:21.379278 3921 ts_tablet_manager.cc:1909] T 19861ddccbe649c096d22fde936bf812 P f575ce92c0224eeb80b9fae5f6561fbe: Deleting consensus metadata
I20240503 02:13:21.384502 29718 catalog_manager.cc:4830] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): tablet 19861ddccbe649c096d22fde936bf812 (table rebalancer_dynamic_table_new_name [id=b11f3b351d8b4d0980f4b390c3bb785f]) successfully deleted
I20240503 02:13:21.385363 3933 tablet_bootstrap.cc:492] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:21.385660 3933 ts_tablet_manager.cc:1367] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.185s user 0.005s sys 0.002s
I20240503 02:13:21.386396 3930 tablet_bootstrap.cc:492] T 3c7789a4948a42a3bd528c629dab7aea P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:13:21.386610 3930 ts_tablet_manager.cc:1367] T 3c7789a4948a42a3bd528c629dab7aea P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.103s user 0.003s sys 0.000s
I20240503 02:13:21.387338 3930 raft_consensus.cc:348] T 3c7789a4948a42a3bd528c629dab7aea P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:21.387614 3930 raft_consensus.cc:374] T 3c7789a4948a42a3bd528c629dab7aea P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:21.387727 3930 raft_consensus.cc:729] T 3c7789a4948a42a3bd528c629dab7aea P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:13:21.388011 3930 consensus_queue.cc:260] T 3c7789a4948a42a3bd528c629dab7aea P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:21.388777 3930 ts_tablet_manager.cc:1398] T 3c7789a4948a42a3bd528c629dab7aea P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:13:21.400624 3933 raft_consensus.cc:348] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:21.400861 3933 raft_consensus.cc:374] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:21.400935 3933 raft_consensus.cc:729] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:21.401166 3933 consensus_queue.cc:260] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:21.404333 3933 ts_tablet_manager.cc:1398] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:13:21.409277 3933 tablet_bootstrap.cc:492] T 9b3b9d83e4c742bfa62624676cbb6a11 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:21.410984 3933 tablet_bootstrap.cc:654] T 9b3b9d83e4c742bfa62624676cbb6a11 P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:21.452944 3933 tablet_bootstrap.cc:492] T 9b3b9d83e4c742bfa62624676cbb6a11 P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:21.453197 3933 ts_tablet_manager.cc:1367] T 9b3b9d83e4c742bfa62624676cbb6a11 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.002s
I20240503 02:13:21.469535 3933 raft_consensus.cc:348] T 9b3b9d83e4c742bfa62624676cbb6a11 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:21.479939 3933 raft_consensus.cc:374] T 9b3b9d83e4c742bfa62624676cbb6a11 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:21.480257 3933 raft_consensus.cc:729] T 9b3b9d83e4c742bfa62624676cbb6a11 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:21.481266 3933 consensus_queue.cc:260] T 9b3b9d83e4c742bfa62624676cbb6a11 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:21.482975 3933 ts_tablet_manager.cc:1398] T 9b3b9d83e4c742bfa62624676cbb6a11 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.029s user 0.000s sys 0.002s
I20240503 02:13:21.497947 29730 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:13:21.524416 3785 raft_consensus.cc:482] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:21.524603 3785 raft_consensus.cc:504] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:21.525327 3785 leader_election.cc:288] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:13:21.526867 566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48b76d7755294d37860842a4a73ab28b" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:21.527086 566 raft_consensus.cc:2454] T 48b76d7755294d37860842a4a73ab28b P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 0.
I20240503 02:13:21.527441 30132 raft_consensus.cc:3037] T 3c7789a4948a42a3bd528c629dab7aea P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:21.529158 30135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48b76d7755294d37860842a4a73ab28b" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:13:21.529327 30135 raft_consensus.cc:2454] T 48b76d7755294d37860842a4a73ab28b P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 0.
I20240503 02:13:21.530870 370 leader_election.cc:302] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f575ce92c0224eeb80b9fae5f6561fbe, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:21.531482 3793 raft_consensus.cc:2781] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:21.531569 3793 raft_consensus.cc:482] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:21.531659 3793 raft_consensus.cc:3037] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:21.535658 433 raft_consensus.cc:1261] T 3c7789a4948a42a3bd528c629dab7aea P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:21.536825 3821 consensus_queue.cc:1035] T 3c7789a4948a42a3bd528c629dab7aea P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:21.535030 3793 raft_consensus.cc:504] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:21.539144 30132 raft_consensus.cc:1261] T 3c7789a4948a42a3bd528c629dab7aea P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:21.539826 3795 consensus_queue.cc:1035] T 3c7789a4948a42a3bd528c629dab7aea P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:21.552577 3794 raft_consensus.cc:482] T 9b3b9d83e4c742bfa62624676cbb6a11 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:21.552776 2947 raft_consensus.cc:482] T 1b6cda34c02e4d24a126171c0063a7d6 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:21.552824 3794 raft_consensus.cc:504] T 9b3b9d83e4c742bfa62624676cbb6a11 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:21.552932 2947 raft_consensus.cc:504] T 1b6cda34c02e4d24a126171c0063a7d6 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:21.553664 3794 leader_election.cc:288] T 9b3b9d83e4c742bfa62624676cbb6a11 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:21.553761 2947 leader_election.cc:288] T 1b6cda34c02e4d24a126171c0063a7d6 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:13:21.554481 30133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b3b9d83e4c742bfa62624676cbb6a11" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:13:21.554551 30135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b6cda34c02e4d24a126171c0063a7d6" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:13:21.554651 30133 raft_consensus.cc:2454] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 0.
I20240503 02:13:21.555297 32638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b6cda34c02e4d24a126171c0063a7d6" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:13:21.555536 32638 raft_consensus.cc:2379] T 1b6cda34c02e4d24a126171c0063a7d6 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 16440e365430405fb10366ab70e2b429 in current term 1: Already voted for candidate a9094240860e43bfb129193f3bfd5be6 in this term.
I20240503 02:13:21.555603 32573 leader_election.cc:302] T 9b3b9d83e4c742bfa62624676cbb6a11 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:13:21.556036 29809 leader_election.cc:302] T 1b6cda34c02e4d24a126171c0063a7d6 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 16440e365430405fb10366ab70e2b429; no voters: 9a4d92bc25374fb3960fde7d10be309d, a9094240860e43bfb129193f3bfd5be6
I20240503 02:13:21.557466 3498 raft_consensus.cc:3037] T 1b6cda34c02e4d24a126171c0063a7d6 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:21.560133 3498 raft_consensus.cc:2726] T 1b6cda34c02e4d24a126171c0063a7d6 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:21.563176 3794 raft_consensus.cc:2781] T 9b3b9d83e4c742bfa62624676cbb6a11 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:21.563318 3794 raft_consensus.cc:482] T 9b3b9d83e4c742bfa62624676cbb6a11 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:21.563406 3794 raft_consensus.cc:3037] T 9b3b9d83e4c742bfa62624676cbb6a11 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:21.560089 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b3b9d83e4c742bfa62624676cbb6a11" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:13:21.576249 436 raft_consensus.cc:2454] T 9b3b9d83e4c742bfa62624676cbb6a11 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 0.
I20240503 02:13:21.576592 3793 leader_election.cc:288] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Requested vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:13:21.577544 566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48b76d7755294d37860842a4a73ab28b" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:13:21.577783 566 raft_consensus.cc:3037] T 48b76d7755294d37860842a4a73ab28b P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:21.581164 566 raft_consensus.cc:2454] T 48b76d7755294d37860842a4a73ab28b P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 1.
I20240503 02:13:21.584476 370 leader_election.cc:302] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f575ce92c0224eeb80b9fae5f6561fbe, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:21.585928 3794 raft_consensus.cc:504] T 9b3b9d83e4c742bfa62624676cbb6a11 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:21.587815 3956 raft_consensus.cc:2781] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:21.587944 3956 raft_consensus.cc:686] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Becoming Leader. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Running, Role: LEADER
I20240503 02:13:21.588228 3956 consensus_queue.cc:237] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:21.595681 3794 leader_election.cc:288] T 9b3b9d83e4c742bfa62624676cbb6a11 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 election: Requested vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:21.596168 30135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b3b9d83e4c742bfa62624676cbb6a11" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:13:21.596406 30135 raft_consensus.cc:3037] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:21.596632 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b3b9d83e4c742bfa62624676cbb6a11" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:13:21.597340 436 raft_consensus.cc:3037] T 9b3b9d83e4c742bfa62624676cbb6a11 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:21.599876 30135 raft_consensus.cc:2454] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 1.
I20240503 02:13:21.600841 32573 leader_election.cc:302] T 9b3b9d83e4c742bfa62624676cbb6a11 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:13:21.601269 3794 raft_consensus.cc:2781] T 9b3b9d83e4c742bfa62624676cbb6a11 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:21.601423 3794 raft_consensus.cc:686] T 9b3b9d83e4c742bfa62624676cbb6a11 P 9a4d92bc25374fb3960fde7d10be309d [term 1 LEADER]: Becoming Leader. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Running, Role: LEADER
I20240503 02:13:21.601809 3794 consensus_queue.cc:237] T 9b3b9d83e4c742bfa62624676cbb6a11 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:21.607405 436 raft_consensus.cc:2454] T 9b3b9d83e4c742bfa62624676cbb6a11 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 1.
I20240503 02:13:21.609588 29730 catalog_manager.cc:5576] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241) does not have the latest schema for tablet 3c7789a4948a42a3bd528c629dab7aea (table rebalancer_dynamic_table [id=f6d702a532b943d5922145591dbc9474]). Expected version 1 got 0
I20240503 02:13:21.613817 29730 catalog_manager.cc:5484] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7). New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:21.614884 30135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48b76d7755294d37860842a4a73ab28b" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:13:21.615068 30135 raft_consensus.cc:3037] T 48b76d7755294d37860842a4a73ab28b P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:21.618291 30135 raft_consensus.cc:2454] T 48b76d7755294d37860842a4a73ab28b P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 1.
I20240503 02:13:21.634603 29730 catalog_manager.cc:5576] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679) does not have the latest schema for tablet 3c7789a4948a42a3bd528c629dab7aea (table rebalancer_dynamic_table [id=f6d702a532b943d5922145591dbc9474]). Expected version 1 got 0
I20240503 02:13:21.642359 3957 mvcc.cc:205] Tried to move back new op lower bound from 7023421036644593664 to 7023421035834482688. Current Snapshot: MvccSnapshot[applied={T|T < 7023421036644593664}]
I20240503 02:13:21.643471 29730 catalog_manager.cc:5484] T 9b3b9d83e4c742bfa62624676cbb6a11 P 9a4d92bc25374fb3960fde7d10be309d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5). New cstate: current_term: 1 leader_uuid: "9a4d92bc25374fb3960fde7d10be309d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:21.650612 3971 tablet.cc:1722] T 3c7789a4948a42a3bd528c629dab7aea P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:21.654577 3973 tablet.cc:1722] T 3c7789a4948a42a3bd528c629dab7aea P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:21.656689 3973 tablet.cc:1620] T 3c7789a4948a42a3bd528c629dab7aea P fbbc2be6382e47908ecb7c68c367bd26: MemRowSet was empty: no flush needed.
I20240503 02:13:21.658143 29730 catalog_manager.cc:5576] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241) does not have the latest schema for tablet 3c7789a4948a42a3bd528c629dab7aea (table rebalancer_dynamic_table [id=f6d702a532b943d5922145591dbc9474]). Expected version 1 got 0
I20240503 02:13:21.651434 3971 tablet.cc:1620] T 3c7789a4948a42a3bd528c629dab7aea P 8a666e798fe8477d8dd1e51b385d5ead: MemRowSet was empty: no flush needed.
I20240503 02:13:21.660553 3972 tablet.cc:1722] T 3c7789a4948a42a3bd528c629dab7aea P a9094240860e43bfb129193f3bfd5be6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:21.661245 3972 tablet.cc:1620] T 3c7789a4948a42a3bd528c629dab7aea P a9094240860e43bfb129193f3bfd5be6: MemRowSet was empty: no flush needed.
I20240503 02:13:21.663295 29733 catalog_manager.cc:5576] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679) does not have the latest schema for tablet 3c7789a4948a42a3bd528c629dab7aea (table rebalancer_dynamic_table [id=f6d702a532b943d5922145591dbc9474]). Expected version 1 got 0
I20240503 02:13:21.668555 29730 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=f6d702a532b943d5922145591dbc9474] alter complete (version 1)
I20240503 02:13:21.695923 30132 consensus_queue.cc:237] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:21.698127 30265 raft_consensus.cc:1261] T 779a6685fc8c4670a71b51c4493bc2f7 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:21.698238 567 raft_consensus.cc:1261] T 779a6685fc8c4670a71b51c4493bc2f7 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:21.698783 3899 consensus_queue.cc:1035] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:21.699066 3900 consensus_queue.cc:1035] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:21.702029 3900 raft_consensus.cc:2932] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:21.702771 30265 raft_consensus.cc:2932] T 779a6685fc8c4670a71b51c4493bc2f7 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:21.706350 567 raft_consensus.cc:2932] T 779a6685fc8c4670a71b51c4493bc2f7 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:21.710196 29733 catalog_manager.cc:5484] T 779a6685fc8c4670a71b51c4493bc2f7 P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: config changed from index 4 to 5, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:21.715402 29853 tablet_service.cc:1514] Processing DeleteTablet for tablet 779a6685fc8c4670a71b51c4493bc2f7 with delete_type TABLET_DATA_TOMBSTONED (TS 16440e365430405fb10366ab70e2b429 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:58744
I20240503 02:13:21.721541 3991 tablet_replica.cc:330] T 779a6685fc8c4670a71b51c4493bc2f7 P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:13:21.721876 3991 raft_consensus.cc:2229] T 779a6685fc8c4670a71b51c4493bc2f7 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:21.722198 3991 raft_consensus.cc:2258] T 779a6685fc8c4670a71b51c4493bc2f7 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:21.722880 3991 ts_tablet_manager.cc:1875] T 779a6685fc8c4670a71b51c4493bc2f7 P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:21.727468 3784 consensus_queue.cc:1035] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:21.731587 3991 ts_tablet_manager.cc:1888] T 779a6685fc8c4670a71b51c4493bc2f7 P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:13:21.731706 3991 log.cc:1196] T 779a6685fc8c4670a71b51c4493bc2f7 P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/779a6685fc8c4670a71b51c4493bc2f7
I20240503 02:13:21.733386 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet 779a6685fc8c4670a71b51c4493bc2f7 (table rebalance_test_table_17 [id=6fdcf94cbd1844948615de6c2a4d18cc]) successfully deleted
I20240503 02:13:21.735862 3784 consensus_queue.cc:1035] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:21.771852 29731 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:13:21.780004 3971 tablet.cc:1722] T 3c7789a4948a42a3bd528c629dab7aea P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:21.780529 3971 tablet.cc:1620] T 3c7789a4948a42a3bd528c629dab7aea P 8a666e798fe8477d8dd1e51b385d5ead: MemRowSet was empty: no flush needed.
I20240503 02:13:21.781912 3972 tablet.cc:1722] T 3c7789a4948a42a3bd528c629dab7aea P a9094240860e43bfb129193f3bfd5be6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:21.782328 3972 tablet.cc:1620] T 3c7789a4948a42a3bd528c629dab7aea P a9094240860e43bfb129193f3bfd5be6: MemRowSet was empty: no flush needed.
I20240503 02:13:21.785508 3973 tablet.cc:1722] T 3c7789a4948a42a3bd528c629dab7aea P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:21.785975 3973 tablet.cc:1620] T 3c7789a4948a42a3bd528c629dab7aea P fbbc2be6382e47908ecb7c68c367bd26: MemRowSet was empty: no flush needed.
I20240503 02:13:21.798942 29731 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=f6d702a532b943d5922145591dbc9474] alter complete (version 2)
I20240503 02:13:21.906203 29729 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:13:21.917647 3971 tablet.cc:1722] T 3c7789a4948a42a3bd528c629dab7aea P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:21.919205 3973 tablet.cc:1722] T 3c7789a4948a42a3bd528c629dab7aea P fbbc2be6382e47908ecb7c68c367bd26: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:21.922813 3972 tablet.cc:1722] T 3c7789a4948a42a3bd528c629dab7aea P a9094240860e43bfb129193f3bfd5be6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:21.937719 29734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table_new_name [id=f6d702a532b943d5922145591dbc9474] alter complete (version 3)
I20240503 02:13:22.030133 3794 consensus_queue.cc:1035] T 9b3b9d83e4c742bfa62624676cbb6a11 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 02:13:22.043596 29731 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:22.043874 29731 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:22.047816 29731 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet 3c7789a4948a42a3bd528c629dab7aea
I20240503 02:13:22.048565 415 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c7789a4948a42a3bd528c629dab7aea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:22 UTC) from {username='slave'} at 127.0.0.1:50814
I20240503 02:13:22.051420 30114 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c7789a4948a42a3bd528c629dab7aea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:22 UTC) from {username='slave'} at 127.0.0.1:44812
I20240503 02:13:22.051467 32754 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c7789a4948a42a3bd528c629dab7aea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:22 UTC) from {username='slave'} at 127.0.0.1:46360
I20240503 02:13:22.080128 4038 tablet_replica.cc:330] T 3c7789a4948a42a3bd528c629dab7aea P a9094240860e43bfb129193f3bfd5be6: stopping tablet replica
I20240503 02:13:22.081205 4038 raft_consensus.cc:2229] T 3c7789a4948a42a3bd528c629dab7aea P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:22.081557 4038 raft_consensus.cc:2258] T 3c7789a4948a42a3bd528c629dab7aea P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:22.083667 4039 tablet_replica.cc:330] T 3c7789a4948a42a3bd528c629dab7aea P 8a666e798fe8477d8dd1e51b385d5ead: stopping tablet replica
I20240503 02:13:22.083928 4039 raft_consensus.cc:2229] T 3c7789a4948a42a3bd528c629dab7aea P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:22.084266 4039 raft_consensus.cc:2258] T 3c7789a4948a42a3bd528c629dab7aea P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:22.084797 4037 tablet_replica.cc:330] T 3c7789a4948a42a3bd528c629dab7aea P fbbc2be6382e47908ecb7c68c367bd26: stopping tablet replica
I20240503 02:13:22.085670 4038 ts_tablet_manager.cc:1875] T 3c7789a4948a42a3bd528c629dab7aea P a9094240860e43bfb129193f3bfd5be6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:22.086232 4037 raft_consensus.cc:2229] T 3c7789a4948a42a3bd528c629dab7aea P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:22.086469 4037 raft_consensus.cc:2258] T 3c7789a4948a42a3bd528c629dab7aea P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:22.088228 4037 ts_tablet_manager.cc:1875] T 3c7789a4948a42a3bd528c629dab7aea P fbbc2be6382e47908ecb7c68c367bd26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:22.088598 3794 consensus_queue.cc:1035] T 9b3b9d83e4c742bfa62624676cbb6a11 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:22.090333 4039 ts_tablet_manager.cc:1875] T 3c7789a4948a42a3bd528c629dab7aea P 8a666e798fe8477d8dd1e51b385d5ead: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:22.105309 4039 ts_tablet_manager.cc:1888] T 3c7789a4948a42a3bd528c629dab7aea P 8a666e798fe8477d8dd1e51b385d5ead: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:22.114852 4037 ts_tablet_manager.cc:1888] T 3c7789a4948a42a3bd528c629dab7aea P fbbc2be6382e47908ecb7c68c367bd26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:22.117709 4039 log.cc:1196] T 3c7789a4948a42a3bd528c629dab7aea P 8a666e798fe8477d8dd1e51b385d5ead: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal/wals/3c7789a4948a42a3bd528c629dab7aea
I20240503 02:13:22.118881 4039 ts_tablet_manager.cc:1909] T 3c7789a4948a42a3bd528c629dab7aea P 8a666e798fe8477d8dd1e51b385d5ead: Deleting consensus metadata
I20240503 02:13:22.120209 4038 ts_tablet_manager.cc:1888] T 3c7789a4948a42a3bd528c629dab7aea P a9094240860e43bfb129193f3bfd5be6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:22.120424 4037 log.cc:1196] T 3c7789a4948a42a3bd528c629dab7aea P fbbc2be6382e47908ecb7c68c367bd26: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal/wals/3c7789a4948a42a3bd528c629dab7aea
I20240503 02:13:22.120391 29719 catalog_manager.cc:4830] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): tablet 3c7789a4948a42a3bd528c629dab7aea (table rebalancer_dynamic_table_new_name [id=f6d702a532b943d5922145591dbc9474]) successfully deleted
I20240503 02:13:22.120481 4038 log.cc:1196] T 3c7789a4948a42a3bd528c629dab7aea P a9094240860e43bfb129193f3bfd5be6: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/wals/3c7789a4948a42a3bd528c629dab7aea
I20240503 02:13:22.120951 4038 ts_tablet_manager.cc:1909] T 3c7789a4948a42a3bd528c629dab7aea P a9094240860e43bfb129193f3bfd5be6: Deleting consensus metadata
I20240503 02:13:22.121047 4037 ts_tablet_manager.cc:1909] T 3c7789a4948a42a3bd528c629dab7aea P fbbc2be6382e47908ecb7c68c367bd26: Deleting consensus metadata
I20240503 02:13:22.122246 29718 catalog_manager.cc:4830] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): tablet 3c7789a4948a42a3bd528c629dab7aea (table rebalancer_dynamic_table_new_name [id=f6d702a532b943d5922145591dbc9474]) successfully deleted
I20240503 02:13:22.123353 29719 catalog_manager.cc:4830] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): tablet 3c7789a4948a42a3bd528c629dab7aea (table rebalancer_dynamic_table_new_name [id=f6d702a532b943d5922145591dbc9474]) successfully deleted
I20240503 02:13:22.131579 3956 consensus_queue.cc:1035] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:22.152794 3956 consensus_queue.cc:1035] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:22.160461 29733 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:13:22.175972 546 tablet_service.cc:1467] Processing CreateTablet for tablet cf9675831ec94f0b8f062498375656ef (DEFAULT_TABLE table=rebalancer_dynamic_table [id=87f01a3a37854860b0e642b017d05736]), partition=
I20240503 02:13:22.176327 30114 tablet_service.cc:1467] Processing CreateTablet for tablet cf9675831ec94f0b8f062498375656ef (DEFAULT_TABLE table=rebalancer_dynamic_table [id=87f01a3a37854860b0e642b017d05736]), partition=
I20240503 02:13:22.176690 30114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf9675831ec94f0b8f062498375656ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:22.176939 546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf9675831ec94f0b8f062498375656ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:22.183306 30244 tablet_service.cc:1467] Processing CreateTablet for tablet cf9675831ec94f0b8f062498375656ef (DEFAULT_TABLE table=rebalancer_dynamic_table [id=87f01a3a37854860b0e642b017d05736]), partition=
I20240503 02:13:22.183674 30244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf9675831ec94f0b8f062498375656ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:22.193465 4047 tablet_bootstrap.cc:492] T cf9675831ec94f0b8f062498375656ef P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:22.197618 4047 tablet_bootstrap.cc:654] T cf9675831ec94f0b8f062498375656ef P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:22.203629 4049 tablet_bootstrap.cc:492] T cf9675831ec94f0b8f062498375656ef P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:13:22.206624 4046 tablet_bootstrap.cc:492] T cf9675831ec94f0b8f062498375656ef P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:13:22.207223 4049 tablet_bootstrap.cc:654] T cf9675831ec94f0b8f062498375656ef P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:22.208168 4047 tablet_bootstrap.cc:492] T cf9675831ec94f0b8f062498375656ef P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:22.208376 4047 ts_tablet_manager.cc:1367] T cf9675831ec94f0b8f062498375656ef P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.005s
I20240503 02:13:22.209343 4047 raft_consensus.cc:348] T cf9675831ec94f0b8f062498375656ef P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:22.210024 4047 raft_consensus.cc:374] T cf9675831ec94f0b8f062498375656ef P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:22.210224 4047 raft_consensus.cc:729] T cf9675831ec94f0b8f062498375656ef P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:22.210974 4046 tablet_bootstrap.cc:654] T cf9675831ec94f0b8f062498375656ef P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:22.211298 4047 consensus_queue.cc:260] T cf9675831ec94f0b8f062498375656ef P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:22.214105 4047 ts_tablet_manager.cc:1398] T cf9675831ec94f0b8f062498375656ef P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:13:22.217425 4049 tablet_bootstrap.cc:492] T cf9675831ec94f0b8f062498375656ef P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:13:22.217669 4049 ts_tablet_manager.cc:1367] T cf9675831ec94f0b8f062498375656ef P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240503 02:13:22.221294 4049 raft_consensus.cc:348] T cf9675831ec94f0b8f062498375656ef P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:22.221668 4049 raft_consensus.cc:374] T cf9675831ec94f0b8f062498375656ef P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:22.221830 4049 raft_consensus.cc:729] T cf9675831ec94f0b8f062498375656ef P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:13:22.222180 4049 consensus_queue.cc:260] T cf9675831ec94f0b8f062498375656ef P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:22.224388 4049 ts_tablet_manager.cc:1398] T cf9675831ec94f0b8f062498375656ef P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:13:22.242307 4046 tablet_bootstrap.cc:492] T cf9675831ec94f0b8f062498375656ef P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:13:22.242570 4046 ts_tablet_manager.cc:1367] T cf9675831ec94f0b8f062498375656ef P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.036s user 0.015s sys 0.005s
I20240503 02:13:22.243309 4046 raft_consensus.cc:348] T cf9675831ec94f0b8f062498375656ef P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:22.243587 4046 raft_consensus.cc:374] T cf9675831ec94f0b8f062498375656ef P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:22.243654 4046 raft_consensus.cc:729] T cf9675831ec94f0b8f062498375656ef P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:13:22.243916 4046 consensus_queue.cc:260] T cf9675831ec94f0b8f062498375656ef P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:22.244778 29984 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:13:22.245162 4046 ts_tablet_manager.cc:1398] T cf9675831ec94f0b8f062498375656ef P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:13:22.252967 3784 raft_consensus.cc:482] T cf9675831ec94f0b8f062498375656ef P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:22.253118 3784 raft_consensus.cc:504] T cf9675831ec94f0b8f062498375656ef P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:22.253800 3784 leader_election.cc:288] T cf9675831ec94f0b8f062498375656ef P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:13:22.255941 566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf9675831ec94f0b8f062498375656ef" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:22.256211 566 raft_consensus.cc:2454] T cf9675831ec94f0b8f062498375656ef P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 0.
I20240503 02:13:22.257587 30069 leader_election.cc:302] T cf9675831ec94f0b8f062498375656ef P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:22.258044 3899 raft_consensus.cc:2781] T cf9675831ec94f0b8f062498375656ef P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:22.258128 3899 raft_consensus.cc:482] T cf9675831ec94f0b8f062498375656ef P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:22.258208 3899 raft_consensus.cc:3037] T cf9675831ec94f0b8f062498375656ef P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:22.260983 3899 raft_consensus.cc:504] T cf9675831ec94f0b8f062498375656ef P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:22.267977 29732 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_extra_table_50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:22.269146 566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf9675831ec94f0b8f062498375656ef" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:13:22.269402 566 raft_consensus.cc:3037] T cf9675831ec94f0b8f062498375656ef P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:22.272343 566 raft_consensus.cc:2454] T cf9675831ec94f0b8f062498375656ef P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 1.
I20240503 02:13:22.272809 30069 leader_election.cc:302] T cf9675831ec94f0b8f062498375656ef P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:22.273113 3784 raft_consensus.cc:2781] T cf9675831ec94f0b8f062498375656ef P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:22.273216 3784 raft_consensus.cc:686] T cf9675831ec94f0b8f062498375656ef P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Becoming Leader. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Running, Role: LEADER
I20240503 02:13:22.273455 3784 consensus_queue.cc:237] T cf9675831ec94f0b8f062498375656ef P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:22.267534 3899 leader_election.cc:288] T cf9675831ec94f0b8f062498375656ef P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Requested vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:13:22.278913 29853 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:22.284636 29984 tablet_service.cc:1467] Processing CreateTablet for tablet 8929e26a53274cecbb808dbd8b203a64 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=e4dd29f93ff64f6088ff5503d1f570dd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:22.285785 30244 tablet_service.cc:1467] Processing CreateTablet for tablet 5faccb32e55d4cb689ff491fd20884e6 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=e4dd29f93ff64f6088ff5503d1f570dd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:22.285981 29853 tablet_service.cc:1467] Processing CreateTablet for tablet 8929e26a53274cecbb808dbd8b203a64 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=e4dd29f93ff64f6088ff5503d1f570dd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:22.286242 30244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5faccb32e55d4cb689ff491fd20884e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:22.286398 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8929e26a53274cecbb808dbd8b203a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:22.289422 32754 tablet_service.cc:1467] Processing CreateTablet for tablet 8929e26a53274cecbb808dbd8b203a64 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=e4dd29f93ff64f6088ff5503d1f570dd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:22.289959 32754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8929e26a53274cecbb808dbd8b203a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:22.291738 29854 tablet_service.cc:1467] Processing CreateTablet for tablet 5faccb32e55d4cb689ff491fd20884e6 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=e4dd29f93ff64f6088ff5503d1f570dd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:22.292145 29854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5faccb32e55d4cb689ff491fd20884e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:22.295606 29984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8929e26a53274cecbb808dbd8b203a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:22.297961 415 tablet_service.cc:1467] Processing CreateTablet for tablet 459c1f3a63b1442191982070ea04cc87 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=e4dd29f93ff64f6088ff5503d1f570dd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:22.298414 415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 459c1f3a63b1442191982070ea04cc87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:22.300045 4049 tablet_bootstrap.cc:492] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:13:22.300304 29986 tablet_service.cc:1467] Processing CreateTablet for tablet 5faccb32e55d4cb689ff491fd20884e6 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=e4dd29f93ff64f6088ff5503d1f570dd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:22.300734 29986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5faccb32e55d4cb689ff491fd20884e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:22.301842 4049 tablet_bootstrap.cc:654] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:22.306764 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf9675831ec94f0b8f062498375656ef" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:13:22.306998 30265 raft_consensus.cc:2454] T cf9675831ec94f0b8f062498375656ef P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 0.
I20240503 02:13:22.307632 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf9675831ec94f0b8f062498375656ef" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:13:22.307835 30265 raft_consensus.cc:3037] T cf9675831ec94f0b8f062498375656ef P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:22.308940 32619 tablet_service.cc:1467] Processing CreateTablet for tablet 459c1f3a63b1442191982070ea04cc87 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=e4dd29f93ff64f6088ff5503d1f570dd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:22.309422 32619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 459c1f3a63b1442191982070ea04cc87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:22.310449 30265 raft_consensus.cc:2454] T cf9675831ec94f0b8f062498375656ef P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 1.
I20240503 02:13:22.328289 29733 catalog_manager.cc:5484] T cf9675831ec94f0b8f062498375656ef P a9094240860e43bfb129193f3bfd5be6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9094240860e43bfb129193f3bfd5be6 (127.22.108.3). New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:22.334297 29985 tablet_service.cc:1467] Processing CreateTablet for tablet 459c1f3a63b1442191982070ea04cc87 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=e4dd29f93ff64f6088ff5503d1f570dd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:22.334744 29985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 459c1f3a63b1442191982070ea04cc87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:22.340596 4069 tablet_bootstrap.cc:492] T 459c1f3a63b1442191982070ea04cc87 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:22.341966 4063 tablet_bootstrap.cc:492] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:22.342263 4049 tablet_bootstrap.cc:492] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:13:22.342475 4049 ts_tablet_manager.cc:1367] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.042s user 0.013s sys 0.008s
I20240503 02:13:22.344044 4049 raft_consensus.cc:348] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:22.344393 4049 raft_consensus.cc:374] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:22.344535 4049 raft_consensus.cc:729] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:13:22.343864 4063 tablet_bootstrap.cc:654] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:22.344837 4049 consensus_queue.cc:260] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:22.352603 4049 ts_tablet_manager.cc:1398] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 02:13:22.355391 4063 tablet_bootstrap.cc:492] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:22.355639 4063 ts_tablet_manager.cc:1367] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.003s
I20240503 02:13:22.357863 4069 tablet_bootstrap.cc:654] T 459c1f3a63b1442191982070ea04cc87 P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:22.356515 4063 raft_consensus.cc:348] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:22.358527 4063 raft_consensus.cc:374] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:22.358628 4063 raft_consensus.cc:729] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:22.358908 4063 consensus_queue.cc:260] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:22.360798 4063 ts_tablet_manager.cc:1398] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 02:13:22.364946 30244 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:13:22.372972 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:22.401755 4073 tablet_bootstrap.cc:492] T 8929e26a53274cecbb808dbd8b203a64 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:22.404654 4069 tablet_bootstrap.cc:492] T 459c1f3a63b1442191982070ea04cc87 P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:22.404942 4069 ts_tablet_manager.cc:1367] T 459c1f3a63b1442191982070ea04cc87 P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.064s user 0.003s sys 0.003s
I20240503 02:13:22.405910 4069 raft_consensus.cc:348] T 459c1f3a63b1442191982070ea04cc87 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:22.406216 4069 raft_consensus.cc:374] T 459c1f3a63b1442191982070ea04cc87 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:22.406368 4069 raft_consensus.cc:729] T 459c1f3a63b1442191982070ea04cc87 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:22.408330 4069 consensus_queue.cc:260] T 459c1f3a63b1442191982070ea04cc87 P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:22.408815 4065 tablet_bootstrap.cc:492] T 8929e26a53274cecbb808dbd8b203a64 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:13:22.409250 4069 ts_tablet_manager.cc:1398] T 459c1f3a63b1442191982070ea04cc87 P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:13:22.412889 4065 tablet_bootstrap.cc:654] T 8929e26a53274cecbb808dbd8b203a64 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:22.419366 32619 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:22.422554 3956 raft_consensus.cc:482] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:22.422749 3956 raft_consensus.cc:504] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:22.423391 3956 leader_election.cc:288] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:13:22.424176 30006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "459c1f3a63b1442191982070ea04cc87" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
W20240503 02:13:22.424793 371 leader_election.cc:341] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:22.425877 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "459c1f3a63b1442191982070ea04cc87" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:13:22.426093 32640 raft_consensus.cc:2454] T 459c1f3a63b1442191982070ea04cc87 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 0.
I20240503 02:13:22.426963 371 leader_election.cc:302] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, fbbc2be6382e47908ecb7c68c367bd26; no voters: 155735c7f8b14997ac4c770aee2a9401
I20240503 02:13:22.427940 3956 raft_consensus.cc:2781] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:22.428038 3956 raft_consensus.cc:482] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:22.428141 3956 raft_consensus.cc:3037] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:22.431403 3956 raft_consensus.cc:504] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:22.435086 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "459c1f3a63b1442191982070ea04cc87" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d"
I20240503 02:13:22.435302 32640 raft_consensus.cc:3037] T 459c1f3a63b1442191982070ea04cc87 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:22.436233 4065 tablet_bootstrap.cc:492] T 8929e26a53274cecbb808dbd8b203a64 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:13:22.436437 4065 ts_tablet_manager.cc:1367] T 8929e26a53274cecbb808dbd8b203a64 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
I20240503 02:13:22.437345 4065 raft_consensus.cc:348] T 8929e26a53274cecbb808dbd8b203a64 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:22.437618 4065 raft_consensus.cc:374] T 8929e26a53274cecbb808dbd8b203a64 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:22.437728 4065 raft_consensus.cc:729] T 8929e26a53274cecbb808dbd8b203a64 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:13:22.438005 4065 consensus_queue.cc:260] T 8929e26a53274cecbb808dbd8b203a64 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:22.438225 30006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "459c1f3a63b1442191982070ea04cc87" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
W20240503 02:13:22.438938 371 leader_election.cc:341] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:22.439116 3956 leader_election.cc:288] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Requested vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:13:22.441305 32640 raft_consensus.cc:2454] T 459c1f3a63b1442191982070ea04cc87 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 1.
I20240503 02:13:22.441752 371 leader_election.cc:302] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, fbbc2be6382e47908ecb7c68c367bd26; no voters: 155735c7f8b14997ac4c770aee2a9401
I20240503 02:13:22.442246 4065 ts_tablet_manager.cc:1398] T 8929e26a53274cecbb808dbd8b203a64 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:13:22.442556 3793 raft_consensus.cc:2781] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:22.442517 29731 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:13:22.444151 3498 raft_consensus.cc:482] T 8929e26a53274cecbb808dbd8b203a64 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:22.445075 3498 raft_consensus.cc:504] T 8929e26a53274cecbb808dbd8b203a64 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:22.445750 3498 leader_election.cc:288] T 8929e26a53274cecbb808dbd8b203a64 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:22.446909 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8929e26a53274cecbb808dbd8b203a64" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
W20240503 02:13:22.448894 29809 leader_election.cc:341] T 8929e26a53274cecbb808dbd8b203a64 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:22.449062 30006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8929e26a53274cecbb808dbd8b203a64" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
W20240503 02:13:22.449564 29809 leader_election.cc:341] T 8929e26a53274cecbb808dbd8b203a64 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:22.449669 29809 leader_election.cc:302] T 8929e26a53274cecbb808dbd8b203a64 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 16440e365430405fb10366ab70e2b429; no voters: 155735c7f8b14997ac4c770aee2a9401, 8a666e798fe8477d8dd1e51b385d5ead
I20240503 02:13:22.449892 3498 raft_consensus.cc:2726] T 8929e26a53274cecbb808dbd8b203a64 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:22.452809 4065 tablet_bootstrap.cc:492] T 5faccb32e55d4cb689ff491fd20884e6 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:13:22.458760 4065 tablet_bootstrap.cc:654] T 5faccb32e55d4cb689ff491fd20884e6 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:22.459905 566 raft_consensus.cc:1261] T cf9675831ec94f0b8f062498375656ef P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:22.460649 3899 consensus_queue.cc:1035] T cf9675831ec94f0b8f062498375656ef P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:22.464920 4074 tablet_bootstrap.cc:492] T 459c1f3a63b1442191982070ea04cc87 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:13:22.477676 30265 raft_consensus.cc:1261] T cf9675831ec94f0b8f062498375656ef P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:22.478281 4074 tablet_bootstrap.cc:654] T 459c1f3a63b1442191982070ea04cc87 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:22.478724 3899 consensus_queue.cc:1035] T cf9675831ec94f0b8f062498375656ef P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:22.412348 4073 tablet_bootstrap.cc:654] T 8929e26a53274cecbb808dbd8b203a64 P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:22.494499 4045 mvcc.cc:205] Tried to move back new op lower bound from 7023421040467750912 to 7023421039714492416. Current Snapshot: MvccSnapshot[applied={T|T < 7023421040467750912}]
I20240503 02:13:22.501161 29732 catalog_manager.cc:5576] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233) does not have the latest schema for tablet cf9675831ec94f0b8f062498375656ef (table rebalancer_dynamic_table [id=87f01a3a37854860b0e642b017d05736]). Expected version 1 got 0
I20240503 02:13:22.506632 4085 tablet.cc:1722] T cf9675831ec94f0b8f062498375656ef P a9094240860e43bfb129193f3bfd5be6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:22.507835 4085 tablet.cc:1620] T cf9675831ec94f0b8f062498375656ef P a9094240860e43bfb129193f3bfd5be6: MemRowSet was empty: no flush needed.
I20240503 02:13:22.514060 4065 tablet_bootstrap.cc:492] T 5faccb32e55d4cb689ff491fd20884e6 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:13:22.514209 4065 ts_tablet_manager.cc:1367] T 5faccb32e55d4cb689ff491fd20884e6 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.061s user 0.003s sys 0.000s
I20240503 02:13:22.515022 4065 raft_consensus.cc:348] T 5faccb32e55d4cb689ff491fd20884e6 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:22.515245 4065 raft_consensus.cc:374] T 5faccb32e55d4cb689ff491fd20884e6 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:22.515303 4065 raft_consensus.cc:729] T 5faccb32e55d4cb689ff491fd20884e6 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:13:22.515712 4065 consensus_queue.cc:260] T 5faccb32e55d4cb689ff491fd20884e6 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:22.519559 29731 catalog_manager.cc:5576] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935) does not have the latest schema for tablet cf9675831ec94f0b8f062498375656ef (table rebalancer_dynamic_table [id=87f01a3a37854860b0e642b017d05736]). Expected version 1 got 0
I20240503 02:13:22.521104 4065 ts_tablet_manager.cc:1398] T 5faccb32e55d4cb689ff491fd20884e6 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:13:22.523417 30114 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:22.526808 3793 raft_consensus.cc:686] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Becoming Leader. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Running, Role: LEADER
I20240503 02:13:22.527110 3793 consensus_queue.cc:237] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:22.535145 4086 tablet.cc:1722] T cf9675831ec94f0b8f062498375656ef P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:22.538061 4086 tablet.cc:1620] T cf9675831ec94f0b8f062498375656ef P f575ce92c0224eeb80b9fae5f6561fbe: MemRowSet was empty: no flush needed.
I20240503 02:13:22.546540 29731 catalog_manager.cc:5484] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7). New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:22.553529 415 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:22.554323 29731 catalog_manager.cc:5576] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233) does not have the latest schema for tablet cf9675831ec94f0b8f062498375656ef (table rebalancer_dynamic_table [id=87f01a3a37854860b0e642b017d05736]). Expected version 1 got 0
I20240503 02:13:22.555990 29732 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=87f01a3a37854860b0e642b017d05736] alter complete (version 1)
I20240503 02:13:22.574594 3811 raft_consensus.cc:482] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:22.574785 3811 raft_consensus.cc:504] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:22.575839 29873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5faccb32e55d4cb689ff491fd20884e6" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:13:22.576042 29873 raft_consensus.cc:2454] T 5faccb32e55d4cb689ff491fd20884e6 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:13:22.576081 4074 tablet_bootstrap.cc:492] T 459c1f3a63b1442191982070ea04cc87 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:13:22.576221 4074 ts_tablet_manager.cc:1367] T 459c1f3a63b1442191982070ea04cc87 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.111s user 0.003s sys 0.000s
I20240503 02:13:22.576524 30202 leader_election.cc:302] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:13:22.576771 3812 raft_consensus.cc:2781] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:22.576853 3812 raft_consensus.cc:482] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:22.576922 3812 raft_consensus.cc:3037] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:22.579617 3812 raft_consensus.cc:504] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:22.580226 4074 raft_consensus.cc:348] T 459c1f3a63b1442191982070ea04cc87 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:22.580469 4074 raft_consensus.cc:374] T 459c1f3a63b1442191982070ea04cc87 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:22.580529 4074 raft_consensus.cc:729] T 459c1f3a63b1442191982070ea04cc87 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:13:22.581080 3812 leader_election.cc:288] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Requested vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:13:22.581394 4074 consensus_queue.cc:260] T 459c1f3a63b1442191982070ea04cc87 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:22.582120 29873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5faccb32e55d4cb689ff491fd20884e6" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:13:22.582298 29873 raft_consensus.cc:3037] T 5faccb32e55d4cb689ff491fd20884e6 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:22.584985 29873 raft_consensus.cc:2454] T 5faccb32e55d4cb689ff491fd20884e6 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:13:22.585697 30202 leader_election.cc:302] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:13:22.586114 3811 leader_election.cc:288] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:13:22.586295 3811 raft_consensus.cc:2781] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:22.586455 3811 raft_consensus.cc:686] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Becoming Leader. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Running, Role: LEADER
I20240503 02:13:22.586751 3811 consensus_queue.cc:237] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:22.587492 4074 ts_tablet_manager.cc:1398] T 459c1f3a63b1442191982070ea04cc87 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:13:22.593271 4074 tablet_bootstrap.cc:492] T 5faccb32e55d4cb689ff491fd20884e6 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:13:22.598220 4074 tablet_bootstrap.cc:654] T 5faccb32e55d4cb689ff491fd20884e6 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:22.602865 30006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5faccb32e55d4cb689ff491fd20884e6" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:13:22.603664 30006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5faccb32e55d4cb689ff491fd20884e6" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
W20240503 02:13:22.605083 30200 leader_election.cc:341] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:13:22.605448 30200 leader_election.cc:341] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:22.609424 4074 tablet_bootstrap.cc:492] T 5faccb32e55d4cb689ff491fd20884e6 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:13:22.609622 4074 ts_tablet_manager.cc:1367] T 5faccb32e55d4cb689ff491fd20884e6 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20240503 02:13:22.610378 4074 raft_consensus.cc:348] T 5faccb32e55d4cb689ff491fd20884e6 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:22.610646 4074 raft_consensus.cc:374] T 5faccb32e55d4cb689ff491fd20884e6 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:22.610755 4074 raft_consensus.cc:729] T 5faccb32e55d4cb689ff491fd20884e6 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:13:22.611061 4074 consensus_queue.cc:260] T 5faccb32e55d4cb689ff491fd20884e6 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:22.616494 4074 ts_tablet_manager.cc:1398] T 5faccb32e55d4cb689ff491fd20884e6 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 02:13:22.616595 4090 tablet.cc:1722] T cf9675831ec94f0b8f062498375656ef P 1f10f80f13c84e80b03eda7085a3b012: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:22.619385 4073 tablet_bootstrap.cc:492] T 8929e26a53274cecbb808dbd8b203a64 P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:22.619621 4073 ts_tablet_manager.cc:1367] T 8929e26a53274cecbb808dbd8b203a64 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.218s user 0.006s sys 0.000s
I20240503 02:13:22.619901 4090 tablet.cc:1620] T cf9675831ec94f0b8f062498375656ef P 1f10f80f13c84e80b03eda7085a3b012: MemRowSet was empty: no flush needed.
I20240503 02:13:22.622007 4074 tablet_bootstrap.cc:492] T 8929e26a53274cecbb808dbd8b203a64 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:13:22.623771 4074 tablet_bootstrap.cc:654] T 8929e26a53274cecbb808dbd8b203a64 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:22.627956 29731 catalog_manager.cc:5576] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935) does not have the latest schema for tablet cf9675831ec94f0b8f062498375656ef (table rebalancer_dynamic_table [id=87f01a3a37854860b0e642b017d05736]). Expected version 1 got 0
I20240503 02:13:22.630856 4073 raft_consensus.cc:348] T 8929e26a53274cecbb808dbd8b203a64 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:22.631276 4073 raft_consensus.cc:374] T 8929e26a53274cecbb808dbd8b203a64 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:22.631445 4073 raft_consensus.cc:729] T 8929e26a53274cecbb808dbd8b203a64 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:22.631808 4073 consensus_queue.cc:260] T 8929e26a53274cecbb808dbd8b203a64 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:22.634831 4073 ts_tablet_manager.cc:1398] T 8929e26a53274cecbb808dbd8b203a64 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:13:22.637365 546 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:22.647778 29731 catalog_manager.cc:5484] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4). New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:22.672549 29732 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:13:22.674449 4074 tablet_bootstrap.cc:492] T 8929e26a53274cecbb808dbd8b203a64 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:13:22.674602 4074 ts_tablet_manager.cc:1367] T 8929e26a53274cecbb808dbd8b203a64 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.053s user 0.002s sys 0.004s
I20240503 02:13:22.675408 4074 raft_consensus.cc:348] T 8929e26a53274cecbb808dbd8b203a64 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:22.675611 4074 raft_consensus.cc:374] T 8929e26a53274cecbb808dbd8b203a64 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:22.675663 4074 raft_consensus.cc:729] T 8929e26a53274cecbb808dbd8b203a64 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:13:22.675866 4074 consensus_queue.cc:260] T 8929e26a53274cecbb808dbd8b203a64 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:22.677539 4074 ts_tablet_manager.cc:1398] T 8929e26a53274cecbb808dbd8b203a64 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:13:22.685312 4085 tablet.cc:1722] T cf9675831ec94f0b8f062498375656ef P a9094240860e43bfb129193f3bfd5be6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:22.685853 4085 tablet.cc:1620] T cf9675831ec94f0b8f062498375656ef P a9094240860e43bfb129193f3bfd5be6: MemRowSet was empty: no flush needed.
I20240503 02:13:22.687517 3431 raft_consensus.cc:482] T 459c1f3a63b1442191982070ea04cc87 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:22.687646 3431 raft_consensus.cc:504] T 459c1f3a63b1442191982070ea04cc87 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:22.688474 3431 leader_election.cc:288] T 459c1f3a63b1442191982070ea04cc87 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:13:22.689242 32638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "459c1f3a63b1442191982070ea04cc87" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:13:22.689517 32638 raft_consensus.cc:2379] T 459c1f3a63b1442191982070ea04cc87 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 155735c7f8b14997ac4c770aee2a9401 in current term 1: Already voted for candidate fbbc2be6382e47908ecb7c68c367bd26 in this term.
I20240503 02:13:22.689970 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "459c1f3a63b1442191982070ea04cc87" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:13:22.690830 29940 leader_election.cc:302] T 459c1f3a63b1442191982070ea04cc87 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401; no voters: 9a4d92bc25374fb3960fde7d10be309d, fbbc2be6382e47908ecb7c68c367bd26
I20240503 02:13:22.692003 3431 raft_consensus.cc:3037] T 459c1f3a63b1442191982070ea04cc87 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:22.691994 4086 tablet.cc:1722] T cf9675831ec94f0b8f062498375656ef P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:22.692435 4086 tablet.cc:1620] T cf9675831ec94f0b8f062498375656ef P f575ce92c0224eeb80b9fae5f6561fbe: MemRowSet was empty: no flush needed.
I20240503 02:13:22.691309 4090 tablet.cc:1722] T cf9675831ec94f0b8f062498375656ef P 1f10f80f13c84e80b03eda7085a3b012: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:22.693365 4090 tablet.cc:1620] T cf9675831ec94f0b8f062498375656ef P 1f10f80f13c84e80b03eda7085a3b012: MemRowSet was empty: no flush needed.
I20240503 02:13:22.694598 3431 raft_consensus.cc:2726] T 459c1f3a63b1442191982070ea04cc87 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:22.704331 29732 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=87f01a3a37854860b0e642b017d05736] alter complete (version 2)
I20240503 02:13:22.787837 3431 raft_consensus.cc:482] T 8929e26a53274cecbb808dbd8b203a64 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:22.788022 3431 raft_consensus.cc:504] T 8929e26a53274cecbb808dbd8b203a64 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:22.788622 3431 leader_election.cc:288] T 8929e26a53274cecbb808dbd8b203a64 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:22.789183 29873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8929e26a53274cecbb808dbd8b203a64" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:13:22.789413 29873 raft_consensus.cc:2454] T 8929e26a53274cecbb808dbd8b203a64 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:13:22.789422 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8929e26a53274cecbb808dbd8b203a64" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:13:22.789625 305 raft_consensus.cc:2454] T 8929e26a53274cecbb808dbd8b203a64 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 0.
I20240503 02:13:22.789876 29942 leader_election.cc:302] T 8929e26a53274cecbb808dbd8b203a64 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429; no voters:
I20240503 02:13:22.790151 3431 raft_consensus.cc:2781] T 8929e26a53274cecbb808dbd8b203a64 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:22.790234 3431 raft_consensus.cc:482] T 8929e26a53274cecbb808dbd8b203a64 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:22.790308 3431 raft_consensus.cc:3037] T 8929e26a53274cecbb808dbd8b203a64 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:22.792908 3431 raft_consensus.cc:504] T 8929e26a53274cecbb808dbd8b203a64 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:22.793494 3431 leader_election.cc:288] T 8929e26a53274cecbb808dbd8b203a64 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Requested vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:22.793994 29873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8929e26a53274cecbb808dbd8b203a64" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:13:22.794186 29873 raft_consensus.cc:3037] T 8929e26a53274cecbb808dbd8b203a64 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:22.794276 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8929e26a53274cecbb808dbd8b203a64" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:13:22.794492 305 raft_consensus.cc:3037] T 8929e26a53274cecbb808dbd8b203a64 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:22.796350 29873 raft_consensus.cc:2454] T 8929e26a53274cecbb808dbd8b203a64 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:13:22.796823 29942 leader_election.cc:302] T 8929e26a53274cecbb808dbd8b203a64 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429; no voters:
I20240503 02:13:22.797119 3431 raft_consensus.cc:2781] T 8929e26a53274cecbb808dbd8b203a64 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:22.797487 305 raft_consensus.cc:2454] T 8929e26a53274cecbb808dbd8b203a64 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:13:22.798081 3431 raft_consensus.cc:686] T 8929e26a53274cecbb808dbd8b203a64 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Becoming Leader. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Running, Role: LEADER
I20240503 02:13:22.798408 3431 consensus_queue.cc:237] T 8929e26a53274cecbb808dbd8b203a64 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:22.802706 29731 catalog_manager.cc:5484] T 8929e26a53274cecbb808dbd8b203a64 P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: term changed from 0 to 1, leader changed from <none> to 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2). New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:22.806488 29732 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:13:22.814177 4085 tablet.cc:1722] T cf9675831ec94f0b8f062498375656ef P a9094240860e43bfb129193f3bfd5be6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:22.815073 4090 tablet.cc:1722] T cf9675831ec94f0b8f062498375656ef P 1f10f80f13c84e80b03eda7085a3b012: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:22.818681 4086 tablet.cc:1722] T cf9675831ec94f0b8f062498375656ef P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:22.833005 29733 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table_new_name [id=87f01a3a37854860b0e642b017d05736] alter complete (version 3)
I20240503 02:13:22.836117 3887 raft_consensus.cc:482] T 5faccb32e55d4cb689ff491fd20884e6 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:22.836297 3887 raft_consensus.cc:504] T 5faccb32e55d4cb689ff491fd20884e6 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:22.837307 3887 leader_election.cc:288] T 5faccb32e55d4cb689ff491fd20884e6 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:13:22.838517 29872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5faccb32e55d4cb689ff491fd20884e6" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:13:22.838768 29872 raft_consensus.cc:2379] T 5faccb32e55d4cb689ff491fd20884e6 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 155735c7f8b14997ac4c770aee2a9401 in current term 1: Already voted for candidate 1f10f80f13c84e80b03eda7085a3b012 in this term.
I20240503 02:13:22.838924 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5faccb32e55d4cb689ff491fd20884e6" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:13:22.839445 29940 leader_election.cc:302] T 5faccb32e55d4cb689ff491fd20884e6 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401; no voters: 16440e365430405fb10366ab70e2b429, 1f10f80f13c84e80b03eda7085a3b012
I20240503 02:13:22.839843 3431 raft_consensus.cc:3037] T 5faccb32e55d4cb689ff491fd20884e6 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:22.846200 3431 raft_consensus.cc:2726] T 5faccb32e55d4cb689ff491fd20884e6 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:22.915041 3793 consensus_queue.cc:1035] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:22.935885 29732 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:22.936167 29732 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:22.939819 29732 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet cf9675831ec94f0b8f062498375656ef
I20240503 02:13:22.940595 30114 tablet_service.cc:1514] Processing DeleteTablet for tablet cf9675831ec94f0b8f062498375656ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:22 UTC) from {username='slave'} at 127.0.0.1:44812
I20240503 02:13:22.940745 30244 tablet_service.cc:1514] Processing DeleteTablet for tablet cf9675831ec94f0b8f062498375656ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:22 UTC) from {username='slave'} at 127.0.0.1:49856
I20240503 02:13:22.941190 4109 tablet_replica.cc:330] T cf9675831ec94f0b8f062498375656ef P a9094240860e43bfb129193f3bfd5be6: stopping tablet replica
I20240503 02:13:22.941210 546 tablet_service.cc:1514] Processing DeleteTablet for tablet cf9675831ec94f0b8f062498375656ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:22 UTC) from {username='slave'} at 127.0.0.1:39028
I20240503 02:13:22.941432 4109 raft_consensus.cc:2229] T cf9675831ec94f0b8f062498375656ef P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:22.942059 4109 raft_consensus.cc:2258] T cf9675831ec94f0b8f062498375656ef P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:22.943233 4109 ts_tablet_manager.cc:1875] T cf9675831ec94f0b8f062498375656ef P a9094240860e43bfb129193f3bfd5be6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:22.947276 4110 tablet_replica.cc:330] T cf9675831ec94f0b8f062498375656ef P 1f10f80f13c84e80b03eda7085a3b012: stopping tablet replica
I20240503 02:13:22.947949 4110 raft_consensus.cc:2229] T cf9675831ec94f0b8f062498375656ef P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:22.949132 4110 raft_consensus.cc:2258] T cf9675831ec94f0b8f062498375656ef P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:22.950645 4109 ts_tablet_manager.cc:1888] T cf9675831ec94f0b8f062498375656ef P a9094240860e43bfb129193f3bfd5be6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:22.950621 3956 consensus_queue.cc:1035] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:22.950780 4109 log.cc:1196] T cf9675831ec94f0b8f062498375656ef P a9094240860e43bfb129193f3bfd5be6: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/wals/cf9675831ec94f0b8f062498375656ef
I20240503 02:13:22.951630 4109 ts_tablet_manager.cc:1909] T cf9675831ec94f0b8f062498375656ef P a9094240860e43bfb129193f3bfd5be6: Deleting consensus metadata
I20240503 02:13:22.952350 4110 ts_tablet_manager.cc:1875] T cf9675831ec94f0b8f062498375656ef P 1f10f80f13c84e80b03eda7085a3b012: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:22.952922 29718 catalog_manager.cc:4830] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): tablet cf9675831ec94f0b8f062498375656ef (table rebalancer_dynamic_table_new_name [id=87f01a3a37854860b0e642b017d05736]) successfully deleted
I20240503 02:13:22.958788 4111 tablet_replica.cc:330] T cf9675831ec94f0b8f062498375656ef P f575ce92c0224eeb80b9fae5f6561fbe: stopping tablet replica
I20240503 02:13:22.959105 4111 raft_consensus.cc:2229] T cf9675831ec94f0b8f062498375656ef P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:22.959439 4111 raft_consensus.cc:2258] T cf9675831ec94f0b8f062498375656ef P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:22.960776 4111 ts_tablet_manager.cc:1875] T cf9675831ec94f0b8f062498375656ef P f575ce92c0224eeb80b9fae5f6561fbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:22.979687 4111 ts_tablet_manager.cc:1888] T cf9675831ec94f0b8f062498375656ef P f575ce92c0224eeb80b9fae5f6561fbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:22.979868 4111 log.cc:1196] T cf9675831ec94f0b8f062498375656ef P f575ce92c0224eeb80b9fae5f6561fbe: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal/wals/cf9675831ec94f0b8f062498375656ef
I20240503 02:13:22.980378 4111 ts_tablet_manager.cc:1909] T cf9675831ec94f0b8f062498375656ef P f575ce92c0224eeb80b9fae5f6561fbe: Deleting consensus metadata
I20240503 02:13:22.985244 4110 ts_tablet_manager.cc:1888] T cf9675831ec94f0b8f062498375656ef P 1f10f80f13c84e80b03eda7085a3b012: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:22.985420 4110 log.cc:1196] T cf9675831ec94f0b8f062498375656ef P 1f10f80f13c84e80b03eda7085a3b012: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/wals/cf9675831ec94f0b8f062498375656ef
I20240503 02:13:22.985888 4110 ts_tablet_manager.cc:1909] T cf9675831ec94f0b8f062498375656ef P 1f10f80f13c84e80b03eda7085a3b012: Deleting consensus metadata
I20240503 02:13:22.987119 29719 catalog_manager.cc:4830] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): tablet cf9675831ec94f0b8f062498375656ef (table rebalancer_dynamic_table_new_name [id=87f01a3a37854860b0e642b017d05736]) successfully deleted
I20240503 02:13:22.988835 29718 catalog_manager.cc:4830] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): tablet cf9675831ec94f0b8f062498375656ef (table rebalancer_dynamic_table_new_name [id=87f01a3a37854860b0e642b017d05736]) successfully deleted
I20240503 02:13:23.022135 29872 consensus_queue.cc:237] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
W20240503 02:13:23.034281 29809 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 493c813acc27492fb952791cf2dc12a2. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:23.035138 30004 raft_consensus.cc:1261] T 493c813acc27492fb952791cf2dc12a2 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:23.035420 566 raft_consensus.cc:1261] T 493c813acc27492fb952791cf2dc12a2 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:23.035732 3498 consensus_queue.cc:1035] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:23.036000 4099 consensus_queue.cc:1035] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:23.039343 3498 raft_consensus.cc:2932] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:23.041481 566 raft_consensus.cc:2932] T 493c813acc27492fb952791cf2dc12a2 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:23.043759 30004 raft_consensus.cc:2932] T 493c813acc27492fb952791cf2dc12a2 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:23.044217 29732 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:13:23.049595 29731 catalog_manager.cc:5484] T 493c813acc27492fb952791cf2dc12a2 P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: config changed from index -1 to 2, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:23.054250 30114 tablet_service.cc:1467] Processing CreateTablet for tablet f203eb4a88544e9aa778694964c51522 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b9cf1367dc464a5692bd5b989bcc627e]), partition=
I20240503 02:13:23.054299 32754 tablet_service.cc:1467] Processing CreateTablet for tablet f203eb4a88544e9aa778694964c51522 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b9cf1367dc464a5692bd5b989bcc627e]), partition=
I20240503 02:13:23.054606 30114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f203eb4a88544e9aa778694964c51522. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:23.054658 32754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f203eb4a88544e9aa778694964c51522. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:23.060736 4073 tablet_bootstrap.cc:492] T f203eb4a88544e9aa778694964c51522 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:23.060894 546 tablet_service.cc:1467] Processing CreateTablet for tablet f203eb4a88544e9aa778694964c51522 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b9cf1367dc464a5692bd5b989bcc627e]), partition=
I20240503 02:13:23.061293 546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f203eb4a88544e9aa778694964c51522. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:23.062701 4073 tablet_bootstrap.cc:654] T f203eb4a88544e9aa778694964c51522 P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:23.066491 4073 tablet_bootstrap.cc:492] T f203eb4a88544e9aa778694964c51522 P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:23.066679 4073 ts_tablet_manager.cc:1367] T f203eb4a88544e9aa778694964c51522 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 02:13:23.067463 4073 raft_consensus.cc:348] T f203eb4a88544e9aa778694964c51522 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:23.067703 4073 raft_consensus.cc:374] T f203eb4a88544e9aa778694964c51522 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:23.067868 4073 raft_consensus.cc:729] T f203eb4a88544e9aa778694964c51522 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:23.068150 4073 consensus_queue.cc:260] T f203eb4a88544e9aa778694964c51522 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:23.070638 4073 ts_tablet_manager.cc:1398] T f203eb4a88544e9aa778694964c51522 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:13:23.071944 4133 tablet_bootstrap.cc:492] T f203eb4a88544e9aa778694964c51522 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:23.074321 4133 tablet_bootstrap.cc:654] T f203eb4a88544e9aa778694964c51522 P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:23.075066 4131 tablet_bootstrap.cc:492] T f203eb4a88544e9aa778694964c51522 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:13:23.080993 4131 tablet_bootstrap.cc:654] T f203eb4a88544e9aa778694964c51522 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:23.086210 4133 tablet_bootstrap.cc:492] T f203eb4a88544e9aa778694964c51522 P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:23.086333 4133 ts_tablet_manager.cc:1367] T f203eb4a88544e9aa778694964c51522 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.009s
I20240503 02:13:23.086791 4131 tablet_bootstrap.cc:492] T f203eb4a88544e9aa778694964c51522 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:13:23.086930 4133 raft_consensus.cc:348] T f203eb4a88544e9aa778694964c51522 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:23.087013 4131 ts_tablet_manager.cc:1367] T f203eb4a88544e9aa778694964c51522 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20240503 02:13:23.087102 4133 raft_consensus.cc:374] T f203eb4a88544e9aa778694964c51522 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:23.087185 4133 raft_consensus.cc:729] T f203eb4a88544e9aa778694964c51522 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:23.087399 4133 consensus_queue.cc:260] T f203eb4a88544e9aa778694964c51522 P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:23.087965 4133 ts_tablet_manager.cc:1398] T f203eb4a88544e9aa778694964c51522 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:13:23.088193 4131 raft_consensus.cc:348] T f203eb4a88544e9aa778694964c51522 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:23.088474 4131 raft_consensus.cc:374] T f203eb4a88544e9aa778694964c51522 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:23.088570 4131 raft_consensus.cc:729] T f203eb4a88544e9aa778694964c51522 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:13:23.088804 4131 consensus_queue.cc:260] T f203eb4a88544e9aa778694964c51522 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:23.090947 4131 ts_tablet_manager.cc:1398] T f203eb4a88544e9aa778694964c51522 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:13:23.103344 3811 consensus_queue.cc:1035] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:23.105639 4091 consensus_queue.cc:1035] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:23.204918 29872 consensus_queue.cc:237] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:23.205070 4096 raft_consensus.cc:482] T f203eb4a88544e9aa778694964c51522 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:23.205221 4096 raft_consensus.cc:504] T f203eb4a88544e9aa778694964c51522 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:23.205962 4096 leader_election.cc:288] T f203eb4a88544e9aa778694964c51522 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:13:23.206384 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f203eb4a88544e9aa778694964c51522" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:13:23.206485 30133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f203eb4a88544e9aa778694964c51522" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:13:23.206601 305 raft_consensus.cc:2454] T f203eb4a88544e9aa778694964c51522 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 0.
I20240503 02:13:23.206677 30133 raft_consensus.cc:2454] T f203eb4a88544e9aa778694964c51522 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 0.
I20240503 02:13:23.207017 502 leader_election.cc:302] T f203eb4a88544e9aa778694964c51522 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:23.207285 4096 raft_consensus.cc:2781] T f203eb4a88544e9aa778694964c51522 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:23.207423 4096 raft_consensus.cc:482] T f203eb4a88544e9aa778694964c51522 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:23.207516 4096 raft_consensus.cc:3037] T f203eb4a88544e9aa778694964c51522 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:23.210229 4096 raft_consensus.cc:504] T f203eb4a88544e9aa778694964c51522 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:23.210880 4096 leader_election.cc:288] T f203eb4a88544e9aa778694964c51522 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 election: Requested vote from peers 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:13:23.210935 32640 raft_consensus.cc:1261] T 58e1aaecf47944569296fc1e0780efb5 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:13:23.211457 29809 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429 -> Peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): Couldn't send request to peer 1f10f80f13c84e80b03eda7085a3b012. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 58e1aaecf47944569296fc1e0780efb5. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:23.211537 433 raft_consensus.cc:1261] T 58e1aaecf47944569296fc1e0780efb5 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:23.211630 3498 consensus_queue.cc:1035] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:23.212039 4130 consensus_queue.cc:1035] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:23.212275 30133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f203eb4a88544e9aa778694964c51522" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:13:23.212463 30133 raft_consensus.cc:3037] T f203eb4a88544e9aa778694964c51522 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:23.212575 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f203eb4a88544e9aa778694964c51522" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:13:23.212873 305 raft_consensus.cc:3037] T f203eb4a88544e9aa778694964c51522 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:23.215255 30133 raft_consensus.cc:2454] T f203eb4a88544e9aa778694964c51522 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 1.
I20240503 02:13:23.215701 501 leader_election.cc:302] T f203eb4a88544e9aa778694964c51522 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:23.216038 4096 raft_consensus.cc:2781] T f203eb4a88544e9aa778694964c51522 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:23.216286 4096 raft_consensus.cc:686] T f203eb4a88544e9aa778694964c51522 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 LEADER]: Becoming Leader. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Running, Role: LEADER
I20240503 02:13:23.216506 305 raft_consensus.cc:2454] T f203eb4a88544e9aa778694964c51522 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 1.
I20240503 02:13:23.216604 4096 consensus_queue.cc:237] T f203eb4a88544e9aa778694964c51522 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:23.221367 29732 catalog_manager.cc:5484] T f203eb4a88544e9aa778694964c51522 P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: term changed from 0 to 1, leader changed from <none> to f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8). New cstate: current_term: 1 leader_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:23.226390 3498 raft_consensus.cc:2932] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:23.227030 433 raft_consensus.cc:2932] T 58e1aaecf47944569296fc1e0780efb5 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:23.229980 32640 raft_consensus.cc:2932] T 58e1aaecf47944569296fc1e0780efb5 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:23.231731 29732 catalog_manager.cc:5484] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429 reported cstate change: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:23.268213 30132 consensus_queue.cc:237] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:23.271227 305 raft_consensus.cc:1261] T e4fe8ae88ae843609a66c7a2e6c7b4af P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:23.271657 565 raft_consensus.cc:1261] T e4fe8ae88ae843609a66c7a2e6c7b4af P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:23.272028 4177 consensus_queue.cc:1035] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:13:23.272802 30070 consensus_peers.cc:485] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 -> Peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Couldn't send request to peer 155735c7f8b14997ac4c770aee2a9401. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e4fe8ae88ae843609a66c7a2e6c7b4af. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:23.272816 4177 consensus_queue.cc:1035] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:23.276049 3899 raft_consensus.cc:2932] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:23.276731 565 raft_consensus.cc:2932] T e4fe8ae88ae843609a66c7a2e6c7b4af P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:23.279748 305 raft_consensus.cc:2932] T e4fe8ae88ae843609a66c7a2e6c7b4af P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:23.283571 29732 catalog_manager.cc:5484] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 reported cstate change: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:23.335803 29733 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:13:23.343123 30132 raft_consensus.cc:1261] T f203eb4a88544e9aa778694964c51522 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:23.344079 305 raft_consensus.cc:1261] T f203eb4a88544e9aa778694964c51522 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:23.344697 4097 consensus_queue.cc:1035] T f203eb4a88544e9aa778694964c51522 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:23.344954 4096 consensus_queue.cc:1035] T f203eb4a88544e9aa778694964c51522 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:23.346877 4100 consensus_queue.cc:1035] T 8929e26a53274cecbb808dbd8b203a64 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:23.353319 3887 consensus_queue.cc:1035] T 8929e26a53274cecbb808dbd8b203a64 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:23.353771 4194 mvcc.cc:205] Tried to move back new op lower bound from 7023421044088164352 to 7023421043578048512. Current Snapshot: MvccSnapshot[applied={T|T < 7023421044088164352}]
I20240503 02:13:23.364745 4199 tablet.cc:1722] T f203eb4a88544e9aa778694964c51522 P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:23.367339 29733 catalog_manager.cc:5576] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975) does not have the latest schema for tablet f203eb4a88544e9aa778694964c51522 (table rebalancer_dynamic_table [id=b9cf1367dc464a5692bd5b989bcc627e]). Expected version 1 got 0
I20240503 02:13:23.369834 4199 tablet.cc:1620] T f203eb4a88544e9aa778694964c51522 P 8a666e798fe8477d8dd1e51b385d5ead: MemRowSet was empty: no flush needed.
I20240503 02:13:23.379108 29733 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=b9cf1367dc464a5692bd5b989bcc627e] alter complete (version 1)
I20240503 02:13:23.397913 4198 tablet.cc:1722] T f203eb4a88544e9aa778694964c51522 P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:23.398386 4198 tablet.cc:1620] T f203eb4a88544e9aa778694964c51522 P f575ce92c0224eeb80b9fae5f6561fbe: MemRowSet was empty: no flush needed.
I20240503 02:13:23.403852 29732 catalog_manager.cc:5576] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679) does not have the latest schema for tablet f203eb4a88544e9aa778694964c51522 (table rebalancer_dynamic_table [id=b9cf1367dc464a5692bd5b989bcc627e]). Expected version 1 got 0
I20240503 02:13:23.406263 4206 tablet.cc:1722] T f203eb4a88544e9aa778694964c51522 P a9094240860e43bfb129193f3bfd5be6: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:23.406829 4206 tablet.cc:1620] T f203eb4a88544e9aa778694964c51522 P a9094240860e43bfb129193f3bfd5be6: MemRowSet was empty: no flush needed.
I20240503 02:13:23.411973 29732 catalog_manager.cc:5576] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679) does not have the latest schema for tablet f203eb4a88544e9aa778694964c51522 (table rebalancer_dynamic_table [id=b9cf1367dc464a5692bd5b989bcc627e]). Expected version 1 got 0
I20240503 02:13:23.417246 29733 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_extra_table_52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:23.434813 29853 tablet_service.cc:1467] Processing CreateTablet for tablet 9cd1418eb38d420c9c41d660476eafb0 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=7feb3c0f2dd74037bb83d98b5b0b5520]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:23.435104 546 tablet_service.cc:1467] Processing CreateTablet for tablet 9cd1418eb38d420c9c41d660476eafb0 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=7feb3c0f2dd74037bb83d98b5b0b5520]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:23.435257 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cd1418eb38d420c9c41d660476eafb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:23.435367 32619 tablet_service.cc:1467] Processing CreateTablet for tablet 9cd1418eb38d420c9c41d660476eafb0 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=7feb3c0f2dd74037bb83d98b5b0b5520]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:23.435539 546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cd1418eb38d420c9c41d660476eafb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:23.435827 32619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cd1418eb38d420c9c41d660476eafb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:23.442167 29854 tablet_service.cc:1467] Processing CreateTablet for tablet 88509fbaa62645c58b7ab1a432eb8727 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=7feb3c0f2dd74037bb83d98b5b0b5520]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:23.442597 29854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88509fbaa62645c58b7ab1a432eb8727. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:23.442705 30114 tablet_service.cc:1467] Processing CreateTablet for tablet 88509fbaa62645c58b7ab1a432eb8727 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=7feb3c0f2dd74037bb83d98b5b0b5520]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:23.443106 30114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88509fbaa62645c58b7ab1a432eb8727. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:23.445695 415 tablet_service.cc:1467] Processing CreateTablet for tablet 52a541b1d3db4228bc5a27af521f0b93 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=7feb3c0f2dd74037bb83d98b5b0b5520]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:23.446074 415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52a541b1d3db4228bc5a27af521f0b93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:23.447119 32754 tablet_service.cc:1467] Processing CreateTablet for tablet 88509fbaa62645c58b7ab1a432eb8727 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=7feb3c0f2dd74037bb83d98b5b0b5520]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:23.447544 32754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88509fbaa62645c58b7ab1a432eb8727. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:23.450466 547 tablet_service.cc:1467] Processing CreateTablet for tablet 52a541b1d3db4228bc5a27af521f0b93 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=7feb3c0f2dd74037bb83d98b5b0b5520]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:23.450901 547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52a541b1d3db4228bc5a27af521f0b93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:23.452559 4131 tablet_bootstrap.cc:492] T 88509fbaa62645c58b7ab1a432eb8727 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:13:23.452665 4212 tablet_bootstrap.cc:492] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:23.454223 4133 tablet_bootstrap.cc:492] T 9cd1418eb38d420c9c41d660476eafb0 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:23.454285 4131 tablet_bootstrap.cc:654] T 88509fbaa62645c58b7ab1a432eb8727 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:23.455847 4133 tablet_bootstrap.cc:654] T 9cd1418eb38d420c9c41d660476eafb0 P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:23.463119 4212 tablet_bootstrap.cc:654] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:23.468513 4212 tablet_bootstrap.cc:492] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:23.469471 4212 ts_tablet_manager.cc:1367] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240503 02:13:23.471642 4212 raft_consensus.cc:348] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:23.471976 4131 tablet_bootstrap.cc:492] T 88509fbaa62645c58b7ab1a432eb8727 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:13:23.472118 4131 ts_tablet_manager.cc:1367] T 88509fbaa62645c58b7ab1a432eb8727 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.005s
I20240503 02:13:23.472818 4131 raft_consensus.cc:348] T 88509fbaa62645c58b7ab1a432eb8727 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:23.473001 4131 raft_consensus.cc:374] T 88509fbaa62645c58b7ab1a432eb8727 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:23.473083 4131 raft_consensus.cc:729] T 88509fbaa62645c58b7ab1a432eb8727 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:13:23.474468 29855 tablet_service.cc:1467] Processing CreateTablet for tablet 52a541b1d3db4228bc5a27af521f0b93 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=7feb3c0f2dd74037bb83d98b5b0b5520]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:23.474944 29855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52a541b1d3db4228bc5a27af521f0b93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:23.475036 4212 raft_consensus.cc:374] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:23.475152 4212 raft_consensus.cc:729] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:23.475364 4212 consensus_queue.cc:260] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:23.477643 4212 ts_tablet_manager.cc:1398] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:13:23.485956 29731 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:13:23.486861 4073 tablet_bootstrap.cc:492] T 88509fbaa62645c58b7ab1a432eb8727 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:23.495337 4198 tablet.cc:1722] T f203eb4a88544e9aa778694964c51522 P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:23.495926 4198 tablet.cc:1620] T f203eb4a88544e9aa778694964c51522 P f575ce92c0224eeb80b9fae5f6561fbe: MemRowSet was empty: no flush needed.
I20240503 02:13:23.496434 4206 tablet.cc:1722] T f203eb4a88544e9aa778694964c51522 P a9094240860e43bfb129193f3bfd5be6: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:23.496924 4206 tablet.cc:1620] T f203eb4a88544e9aa778694964c51522 P a9094240860e43bfb129193f3bfd5be6: MemRowSet was empty: no flush needed.
I20240503 02:13:23.505412 4073 tablet_bootstrap.cc:654] T 88509fbaa62645c58b7ab1a432eb8727 P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:23.513399 4199 tablet.cc:1722] T f203eb4a88544e9aa778694964c51522 P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:23.513927 4199 tablet.cc:1620] T f203eb4a88544e9aa778694964c51522 P 8a666e798fe8477d8dd1e51b385d5ead: MemRowSet was empty: no flush needed.
I20240503 02:13:23.473311 4131 consensus_queue.cc:260] T 88509fbaa62645c58b7ab1a432eb8727 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:23.520216 4131 ts_tablet_manager.cc:1398] T 88509fbaa62645c58b7ab1a432eb8727 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.048s user 0.001s sys 0.002s
I20240503 02:13:23.520906 4217 tablet_bootstrap.cc:492] T 9cd1418eb38d420c9c41d660476eafb0 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:23.523406 4217 tablet_bootstrap.cc:654] T 9cd1418eb38d420c9c41d660476eafb0 P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:23.528416 29732 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=b9cf1367dc464a5692bd5b989bcc627e] alter complete (version 2)
I20240503 02:13:23.543567 4217 tablet_bootstrap.cc:492] T 9cd1418eb38d420c9c41d660476eafb0 P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:23.543988 4217 ts_tablet_manager.cc:1367] T 9cd1418eb38d420c9c41d660476eafb0 P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.004s
I20240503 02:13:23.545286 4217 raft_consensus.cc:348] T 9cd1418eb38d420c9c41d660476eafb0 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:23.545717 4217 raft_consensus.cc:374] T 9cd1418eb38d420c9c41d660476eafb0 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:23.545898 4217 raft_consensus.cc:729] T 9cd1418eb38d420c9c41d660476eafb0 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:23.546335 4217 consensus_queue.cc:260] T 9cd1418eb38d420c9c41d660476eafb0 P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:23.547389 4217 ts_tablet_manager.cc:1398] T 9cd1418eb38d420c9c41d660476eafb0 P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:23.550864 4216 tablet_bootstrap.cc:492] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:13:23.552120 4073 tablet_bootstrap.cc:492] T 88509fbaa62645c58b7ab1a432eb8727 P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:23.552285 4073 ts_tablet_manager.cc:1367] T 88509fbaa62645c58b7ab1a432eb8727 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.066s user 0.007s sys 0.005s
I20240503 02:13:23.557855 4073 raft_consensus.cc:348] T 88509fbaa62645c58b7ab1a432eb8727 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:23.558074 4073 raft_consensus.cc:374] T 88509fbaa62645c58b7ab1a432eb8727 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:23.558176 4073 raft_consensus.cc:729] T 88509fbaa62645c58b7ab1a432eb8727 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:23.558741 4073 consensus_queue.cc:260] T 88509fbaa62645c58b7ab1a432eb8727 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:23.559398 4073 ts_tablet_manager.cc:1398] T 88509fbaa62645c58b7ab1a432eb8727 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:13:23.562222 4216 tablet_bootstrap.cc:654] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:23.565127 4222 ts_tablet_manager.cc:902] T 493c813acc27492fb952791cf2dc12a2 P 9a4d92bc25374fb3960fde7d10be309d: Initiating tablet copy from peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:13:23.567978 4133 tablet_bootstrap.cc:492] T 9cd1418eb38d420c9c41d660476eafb0 P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:23.568194 4133 ts_tablet_manager.cc:1367] T 9cd1418eb38d420c9c41d660476eafb0 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.114s user 0.010s sys 0.034s
I20240503 02:13:23.568972 4133 raft_consensus.cc:348] T 9cd1418eb38d420c9c41d660476eafb0 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:23.569247 4133 raft_consensus.cc:374] T 9cd1418eb38d420c9c41d660476eafb0 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:23.569365 4133 raft_consensus.cc:729] T 9cd1418eb38d420c9c41d660476eafb0 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:23.569667 4133 consensus_queue.cc:260] T 9cd1418eb38d420c9c41d660476eafb0 P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:23.570413 4133 ts_tablet_manager.cc:1398] T 9cd1418eb38d420c9c41d660476eafb0 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:23.570845 4133 tablet_bootstrap.cc:492] T 52a541b1d3db4228bc5a27af521f0b93 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:23.572621 4133 tablet_bootstrap.cc:654] T 52a541b1d3db4228bc5a27af521f0b93 P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:23.580874 4216 tablet_bootstrap.cc:492] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:13:23.581159 4216 ts_tablet_manager.cc:1367] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.005s
I20240503 02:13:23.582196 4216 raft_consensus.cc:348] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:23.584283 4216 raft_consensus.cc:374] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:23.584508 4216 raft_consensus.cc:729] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:13:23.584568 4133 tablet_bootstrap.cc:492] T 52a541b1d3db4228bc5a27af521f0b93 P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:23.584697 4133 ts_tablet_manager.cc:1367] T 52a541b1d3db4228bc5a27af521f0b93 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.006s
I20240503 02:13:23.585122 4216 consensus_queue.cc:260] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:23.585634 4133 raft_consensus.cc:348] T 52a541b1d3db4228bc5a27af521f0b93 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:23.585837 4133 raft_consensus.cc:374] T 52a541b1d3db4228bc5a27af521f0b93 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:23.585945 4133 raft_consensus.cc:729] T 52a541b1d3db4228bc5a27af521f0b93 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:23.586198 4133 consensus_queue.cc:260] T 52a541b1d3db4228bc5a27af521f0b93 P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:23.589198 4216 ts_tablet_manager.cc:1398] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240503 02:13:23.590821 4216 tablet_bootstrap.cc:492] T 88509fbaa62645c58b7ab1a432eb8727 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:13:23.592090 4133 ts_tablet_manager.cc:1398] T 52a541b1d3db4228bc5a27af521f0b93 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:13:23.595886 4216 tablet_bootstrap.cc:654] T 88509fbaa62645c58b7ab1a432eb8727 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:23.610194 4222 tablet_copy_client.cc:308] T 493c813acc27492fb952791cf2dc12a2 P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.1:36719
I20240503 02:13:23.611243 29895 tablet_copy_service.cc:140] P 16440e365430405fb10366ab70e2b429: Received BeginTabletCopySession request for tablet 493c813acc27492fb952791cf2dc12a2 from peer 9a4d92bc25374fb3960fde7d10be309d ({username='slave'} at 127.22.108.5:49385)
I20240503 02:13:23.611378 29895 tablet_copy_service.cc:161] P 16440e365430405fb10366ab70e2b429: Beginning new tablet copy session on tablet 493c813acc27492fb952791cf2dc12a2 from peer 9a4d92bc25374fb3960fde7d10be309d at {username='slave'} at 127.22.108.5:49385: session id = 9a4d92bc25374fb3960fde7d10be309d-493c813acc27492fb952791cf2dc12a2
I20240503 02:13:23.614244 29895 tablet_copy_source_session.cc:205] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:23.618468 4222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493c813acc27492fb952791cf2dc12a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:23.621428 4216 tablet_bootstrap.cc:492] T 88509fbaa62645c58b7ab1a432eb8727 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:13:23.621624 4216 ts_tablet_manager.cc:1367] T 88509fbaa62645c58b7ab1a432eb8727 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.004s
I20240503 02:13:23.622400 4216 raft_consensus.cc:348] T 88509fbaa62645c58b7ab1a432eb8727 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:23.622601 4216 raft_consensus.cc:374] T 88509fbaa62645c58b7ab1a432eb8727 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:23.623605 4216 raft_consensus.cc:729] T 88509fbaa62645c58b7ab1a432eb8727 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:13:23.623879 4216 consensus_queue.cc:260] T 88509fbaa62645c58b7ab1a432eb8727 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:23.626217 4216 ts_tablet_manager.cc:1398] T 88509fbaa62645c58b7ab1a432eb8727 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 02:13:23.629834 4216 tablet_bootstrap.cc:492] T 52a541b1d3db4228bc5a27af521f0b93 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:13:23.636915 29731 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:13:23.632647 4216 tablet_bootstrap.cc:654] T 52a541b1d3db4228bc5a27af521f0b93 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:23.638155 4222 tablet_copy_client.cc:783] T 493c813acc27492fb952791cf2dc12a2 P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:23.638494 4222 tablet_copy_client.cc:647] T 493c813acc27492fb952791cf2dc12a2 P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:23.647378 4198 tablet.cc:1722] T f203eb4a88544e9aa778694964c51522 P f575ce92c0224eeb80b9fae5f6561fbe: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:23.648466 4199 tablet.cc:1722] T f203eb4a88544e9aa778694964c51522 P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:23.654042 4206 tablet.cc:1722] T f203eb4a88544e9aa778694964c51522 P a9094240860e43bfb129193f3bfd5be6: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:23.655817 4222 tablet_copy_client.cc:523] T 493c813acc27492fb952791cf2dc12a2 P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:23.663731 4222 tablet_bootstrap.cc:492] T 493c813acc27492fb952791cf2dc12a2 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:23.666344 29731 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table_new_name [id=b9cf1367dc464a5692bd5b989bcc627e] alter complete (version 3)
I20240503 02:13:23.667449 4147 raft_consensus.cc:482] T 9cd1418eb38d420c9c41d660476eafb0 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:23.667649 4147 raft_consensus.cc:504] T 9cd1418eb38d420c9c41d660476eafb0 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:23.669409 4216 tablet_bootstrap.cc:492] T 52a541b1d3db4228bc5a27af521f0b93 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:13:23.669571 4216 ts_tablet_manager.cc:1367] T 52a541b1d3db4228bc5a27af521f0b93 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.040s user 0.001s sys 0.002s
I20240503 02:13:23.669854 4147 leader_election.cc:288] T 9cd1418eb38d420c9c41d660476eafb0 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:13:23.671854 565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd1418eb38d420c9c41d660476eafb0" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:23.672084 565 raft_consensus.cc:2454] T 9cd1418eb38d420c9c41d660476eafb0 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 0.
I20240503 02:13:23.673272 32573 leader_election.cc:302] T 9cd1418eb38d420c9c41d660476eafb0 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:23.673545 4147 raft_consensus.cc:2781] T 9cd1418eb38d420c9c41d660476eafb0 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:23.673671 4147 raft_consensus.cc:482] T 9cd1418eb38d420c9c41d660476eafb0 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:23.673760 4147 raft_consensus.cc:3037] T 9cd1418eb38d420c9c41d660476eafb0 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:23.671130 29872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd1418eb38d420c9c41d660476eafb0" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:13:23.675941 29872 raft_consensus.cc:2454] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 0.
I20240503 02:13:23.670446 4216 raft_consensus.cc:348] T 52a541b1d3db4228bc5a27af521f0b93 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:23.676153 4216 raft_consensus.cc:374] T 52a541b1d3db4228bc5a27af521f0b93 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:23.676216 4216 raft_consensus.cc:729] T 52a541b1d3db4228bc5a27af521f0b93 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:13:23.676714 4216 consensus_queue.cc:260] T 52a541b1d3db4228bc5a27af521f0b93 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:23.677335 4216 ts_tablet_manager.cc:1398] T 52a541b1d3db4228bc5a27af521f0b93 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:13:23.678911 4147 raft_consensus.cc:504] T 9cd1418eb38d420c9c41d660476eafb0 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:23.680092 29872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd1418eb38d420c9c41d660476eafb0" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:13:23.680244 29872 raft_consensus.cc:3037] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:23.681255 4147 leader_election.cc:288] T 9cd1418eb38d420c9c41d660476eafb0 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 election: Requested vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:13:23.683041 29872 raft_consensus.cc:2454] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 1.
I20240503 02:13:23.683425 32576 leader_election.cc:302] T 9cd1418eb38d420c9c41d660476eafb0 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, 9a4d92bc25374fb3960fde7d10be309d; no voters:
I20240503 02:13:23.684319 565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd1418eb38d420c9c41d660476eafb0" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:13:23.684484 3794 raft_consensus.cc:2781] T 9cd1418eb38d420c9c41d660476eafb0 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:23.684551 565 raft_consensus.cc:3037] T 9cd1418eb38d420c9c41d660476eafb0 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:23.686125 4240 ts_tablet_manager.cc:902] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012: Initiating tablet copy from peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:13:23.687115 3794 raft_consensus.cc:686] T 9cd1418eb38d420c9c41d660476eafb0 P 9a4d92bc25374fb3960fde7d10be309d [term 1 LEADER]: Becoming Leader. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Running, Role: LEADER
I20240503 02:13:23.687404 3794 consensus_queue.cc:237] T 9cd1418eb38d420c9c41d660476eafb0 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:23.688014 565 raft_consensus.cc:2454] T 9cd1418eb38d420c9c41d660476eafb0 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 1.
I20240503 02:13:23.691990 29733 catalog_manager.cc:5484] T 9cd1418eb38d420c9c41d660476eafb0 P 9a4d92bc25374fb3960fde7d10be309d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5). New cstate: current_term: 1 leader_uuid: "9a4d92bc25374fb3960fde7d10be309d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:23.705760 4078 raft_consensus.cc:482] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:23.705986 4078 raft_consensus.cc:504] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:23.707026 4078 leader_election.cc:288] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:13:23.708638 565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52a541b1d3db4228bc5a27af521f0b93" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:23.708698 29872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52a541b1d3db4228bc5a27af521f0b93" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:13:23.708873 565 raft_consensus.cc:2454] T 52a541b1d3db4228bc5a27af521f0b93 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 0.
I20240503 02:13:23.709481 370 leader_election.cc:302] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f575ce92c0224eeb80b9fae5f6561fbe, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:23.709009 29872 raft_consensus.cc:2454] T 52a541b1d3db4228bc5a27af521f0b93 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 0.
I20240503 02:13:23.710384 3956 raft_consensus.cc:2781] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:23.710489 3956 raft_consensus.cc:482] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:23.712524 3956 raft_consensus.cc:3037] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:23.719784 3956 raft_consensus.cc:504] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:23.723376 565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52a541b1d3db4228bc5a27af521f0b93" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:13:23.723678 565 raft_consensus.cc:3037] T 52a541b1d3db4228bc5a27af521f0b93 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:23.727797 565 raft_consensus.cc:2454] T 52a541b1d3db4228bc5a27af521f0b93 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 1.
I20240503 02:13:23.728397 370 leader_election.cc:302] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f575ce92c0224eeb80b9fae5f6561fbe, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:23.728808 4078 raft_consensus.cc:2781] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:23.730312 29872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52a541b1d3db4228bc5a27af521f0b93" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:13:23.730592 29872 raft_consensus.cc:3037] T 52a541b1d3db4228bc5a27af521f0b93 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:23.732347 4078 raft_consensus.cc:686] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Becoming Leader. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Running, Role: LEADER
I20240503 02:13:23.732908 4078 consensus_queue.cc:237] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:23.734673 29872 raft_consensus.cc:2454] T 52a541b1d3db4228bc5a27af521f0b93 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 1.
I20240503 02:13:23.741513 29733 catalog_manager.cc:5484] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7). New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:23.742578 3956 leader_election.cc:288] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 1 election: Requested vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:13:23.745394 4222 tablet_bootstrap.cc:492] T 493c813acc27492fb952791cf2dc12a2 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:23.746027 4222 tablet_bootstrap.cc:492] T 493c813acc27492fb952791cf2dc12a2 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap complete.
I20240503 02:13:23.746349 4222 ts_tablet_manager.cc:1367] T 493c813acc27492fb952791cf2dc12a2 P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.083s user 0.008s sys 0.002s
I20240503 02:13:23.747560 4222 raft_consensus.cc:348] T 493c813acc27492fb952791cf2dc12a2 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:23.748018 4222 raft_consensus.cc:729] T 493c813acc27492fb952791cf2dc12a2 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:23.748394 4222 consensus_queue.cc:260] T 493c813acc27492fb952791cf2dc12a2 P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:23.749276 4222 ts_tablet_manager.cc:1398] T 493c813acc27492fb952791cf2dc12a2 P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:13:23.750543 29895 tablet_copy_service.cc:342] P 16440e365430405fb10366ab70e2b429: Request end of tablet copy session 9a4d92bc25374fb3960fde7d10be309d-493c813acc27492fb952791cf2dc12a2 received from {username='slave'} at 127.22.108.5:49385
I20240503 02:13:23.750690 29895 tablet_copy_service.cc:433] P 16440e365430405fb10366ab70e2b429: ending tablet copy session 9a4d92bc25374fb3960fde7d10be309d-493c813acc27492fb952791cf2dc12a2 on tablet 493c813acc27492fb952791cf2dc12a2 with peer 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:13:23.771485 4240 tablet_copy_client.cc:308] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.1:36719
I20240503 02:13:23.772194 29895 tablet_copy_service.cc:140] P 16440e365430405fb10366ab70e2b429: Received BeginTabletCopySession request for tablet 58e1aaecf47944569296fc1e0780efb5 from peer 1f10f80f13c84e80b03eda7085a3b012 ({username='slave'} at 127.22.108.4:51273)
I20240503 02:13:23.772384 29895 tablet_copy_service.cc:161] P 16440e365430405fb10366ab70e2b429: Beginning new tablet copy session on tablet 58e1aaecf47944569296fc1e0780efb5 from peer 1f10f80f13c84e80b03eda7085a3b012 at {username='slave'} at 127.22.108.4:51273: session id = 1f10f80f13c84e80b03eda7085a3b012-58e1aaecf47944569296fc1e0780efb5
I20240503 02:13:23.775776 29895 tablet_copy_source_session.cc:205] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:23.776083 29732 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:23.776270 29732 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:23.779227 4240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58e1aaecf47944569296fc1e0780efb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:23.780718 29732 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet f203eb4a88544e9aa778694964c51522
I20240503 02:13:23.782343 32754 tablet_service.cc:1514] Processing DeleteTablet for tablet f203eb4a88544e9aa778694964c51522 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:23 UTC) from {username='slave'} at 127.0.0.1:46360
I20240503 02:13:23.784875 547 tablet_service.cc:1514] Processing DeleteTablet for tablet f203eb4a88544e9aa778694964c51522 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:23 UTC) from {username='slave'} at 127.0.0.1:39028
I20240503 02:13:23.785054 30114 tablet_service.cc:1514] Processing DeleteTablet for tablet f203eb4a88544e9aa778694964c51522 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:23 UTC) from {username='slave'} at 127.0.0.1:44812
I20240503 02:13:23.787276 4240 tablet_copy_client.cc:783] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:23.787559 4240 tablet_copy_client.cc:647] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:23.792377 4240 tablet_copy_client.cc:523] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:23.797266 4240 tablet_bootstrap.cc:492] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:13:23.810719 4261 tablet_replica.cc:330] T f203eb4a88544e9aa778694964c51522 P a9094240860e43bfb129193f3bfd5be6: stopping tablet replica
I20240503 02:13:23.811198 4261 raft_consensus.cc:2229] T f203eb4a88544e9aa778694964c51522 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:23.811411 4261 raft_consensus.cc:2258] T f203eb4a88544e9aa778694964c51522 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:23.813541 4261 ts_tablet_manager.cc:1875] T f203eb4a88544e9aa778694964c51522 P a9094240860e43bfb129193f3bfd5be6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:23.820781 4260 tablet_replica.cc:330] T f203eb4a88544e9aa778694964c51522 P f575ce92c0224eeb80b9fae5f6561fbe: stopping tablet replica
I20240503 02:13:23.821707 4260 raft_consensus.cc:2229] T f203eb4a88544e9aa778694964c51522 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:23.822923 4260 raft_consensus.cc:2258] T f203eb4a88544e9aa778694964c51522 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:23.831784 4259 tablet_replica.cc:330] T f203eb4a88544e9aa778694964c51522 P 8a666e798fe8477d8dd1e51b385d5ead: stopping tablet replica
I20240503 02:13:23.833493 4260 ts_tablet_manager.cc:1875] T f203eb4a88544e9aa778694964c51522 P f575ce92c0224eeb80b9fae5f6561fbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:23.833842 4259 raft_consensus.cc:2229] T f203eb4a88544e9aa778694964c51522 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:23.846899 4263 ts_tablet_manager.cc:902] T e4fe8ae88ae843609a66c7a2e6c7b4af P 155735c7f8b14997ac4c770aee2a9401: Initiating tablet copy from peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:13:23.845727 4259 raft_consensus.cc:2258] T f203eb4a88544e9aa778694964c51522 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:23.856786 4259 ts_tablet_manager.cc:1875] T f203eb4a88544e9aa778694964c51522 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:23.883282 29729 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:13:23.892702 29985 tablet_service.cc:1467] Processing CreateTablet for tablet acb04344073b4ceaaa6d404c1a774434 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d5adfab772b6453b980eeb511818c012]), partition=
I20240503 02:13:23.893107 29985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acb04344073b4ceaaa6d404c1a774434. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:23.893061 32754 tablet_service.cc:1467] Processing CreateTablet for tablet acb04344073b4ceaaa6d404c1a774434 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d5adfab772b6453b980eeb511818c012]), partition=
I20240503 02:13:23.894132 32754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acb04344073b4ceaaa6d404c1a774434. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:23.897660 32619 tablet_service.cc:1467] Processing CreateTablet for tablet acb04344073b4ceaaa6d404c1a774434 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d5adfab772b6453b980eeb511818c012]), partition=
I20240503 02:13:23.898051 32619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acb04344073b4ceaaa6d404c1a774434. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:23.906337 4073 tablet_bootstrap.cc:492] T acb04344073b4ceaaa6d404c1a774434 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:23.907999 4073 tablet_bootstrap.cc:654] T acb04344073b4ceaaa6d404c1a774434 P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:23.912786 4219 raft_consensus.cc:482] T 88509fbaa62645c58b7ab1a432eb8727 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:23.912995 4219 raft_consensus.cc:504] T 88509fbaa62645c58b7ab1a432eb8727 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:23.913379 4177 raft_consensus.cc:482] T 88509fbaa62645c58b7ab1a432eb8727 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:23.913537 4177 raft_consensus.cc:504] T 88509fbaa62645c58b7ab1a432eb8727 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:23.913771 4219 leader_election.cc:288] T 88509fbaa62645c58b7ab1a432eb8727 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:13:23.914237 4177 leader_election.cc:288] T 88509fbaa62645c58b7ab1a432eb8727 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:23.915431 4217 tablet_bootstrap.cc:492] T acb04344073b4ceaaa6d404c1a774434 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:23.916189 29873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88509fbaa62645c58b7ab1a432eb8727" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:13:23.916417 29873 raft_consensus.cc:2454] T 88509fbaa62645c58b7ab1a432eb8727 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:13:23.917698 4217 tablet_bootstrap.cc:654] T acb04344073b4ceaaa6d404c1a774434 P 9a4d92bc25374fb3960fde7d10be309d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:23.917913 32711 leader_election.cc:302] T 88509fbaa62645c58b7ab1a432eb8727 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, 8a666e798fe8477d8dd1e51b385d5ead; no voters:
I20240503 02:13:23.918658 29872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88509fbaa62645c58b7ab1a432eb8727" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:13:23.918895 29872 raft_consensus.cc:2454] T 88509fbaa62645c58b7ab1a432eb8727 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 0.
I20240503 02:13:23.919360 306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88509fbaa62645c58b7ab1a432eb8727" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:13:23.919559 306 raft_consensus.cc:2454] T 88509fbaa62645c58b7ab1a432eb8727 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 0.
I20240503 02:13:23.920200 30072 leader_election.cc:302] T 88509fbaa62645c58b7ab1a432eb8727 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:13:23.920976 4177 raft_consensus.cc:2781] T 88509fbaa62645c58b7ab1a432eb8727 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:23.921104 4177 raft_consensus.cc:482] T 88509fbaa62645c58b7ab1a432eb8727 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:23.921182 4177 raft_consensus.cc:3037] T 88509fbaa62645c58b7ab1a432eb8727 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:23.921140 30132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88509fbaa62645c58b7ab1a432eb8727" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:13:23.921615 3795 raft_consensus.cc:2781] T 88509fbaa62645c58b7ab1a432eb8727 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:23.921698 3795 raft_consensus.cc:482] T 88509fbaa62645c58b7ab1a432eb8727 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:23.921782 3795 raft_consensus.cc:3037] T 88509fbaa62645c58b7ab1a432eb8727 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:23.924917 4177 raft_consensus.cc:504] T 88509fbaa62645c58b7ab1a432eb8727 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:23.925997 4177 leader_election.cc:288] T 88509fbaa62645c58b7ab1a432eb8727 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Requested vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:23.926036 30132 raft_consensus.cc:2379] T 88509fbaa62645c58b7ab1a432eb8727 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8a666e798fe8477d8dd1e51b385d5ead in current term 1: Already voted for candidate a9094240860e43bfb129193f3bfd5be6 in this term.
I20240503 02:13:23.924561 3795 raft_consensus.cc:504] T 88509fbaa62645c58b7ab1a432eb8727 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:23.928977 29872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88509fbaa62645c58b7ab1a432eb8727" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:13:23.929230 29872 raft_consensus.cc:3037] T 88509fbaa62645c58b7ab1a432eb8727 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:23.926700 306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88509fbaa62645c58b7ab1a432eb8727" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:13:23.930627 306 raft_consensus.cc:2379] T 88509fbaa62645c58b7ab1a432eb8727 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a9094240860e43bfb129193f3bfd5be6 in current term 1: Already voted for candidate 8a666e798fe8477d8dd1e51b385d5ead in this term.
I20240503 02:13:23.933926 4240 tablet_bootstrap.cc:492] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:23.932108 29872 raft_consensus.cc:2454] T 88509fbaa62645c58b7ab1a432eb8727 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 1.
I20240503 02:13:23.934459 4240 tablet_bootstrap.cc:492] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap complete.
I20240503 02:13:23.934734 4240 ts_tablet_manager.cc:1367] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.138s user 0.033s sys 0.054s
I20240503 02:13:23.935703 30072 leader_election.cc:302] T 88509fbaa62645c58b7ab1a432eb8727 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16440e365430405fb10366ab70e2b429, a9094240860e43bfb129193f3bfd5be6; no voters: 8a666e798fe8477d8dd1e51b385d5ead
I20240503 02:13:23.935667 4240 raft_consensus.cc:348] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:23.935891 4240 raft_consensus.cc:729] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:13:23.935947 29872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88509fbaa62645c58b7ab1a432eb8727" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:13:23.936178 29872 raft_consensus.cc:2379] T 88509fbaa62645c58b7ab1a432eb8727 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a666e798fe8477d8dd1e51b385d5ead in current term 1: Already voted for candidate a9094240860e43bfb129193f3bfd5be6 in this term.
I20240503 02:13:23.936277 4177 raft_consensus.cc:2781] T 88509fbaa62645c58b7ab1a432eb8727 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:23.936398 4240 consensus_queue.cc:260] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:23.936851 4177 raft_consensus.cc:686] T 88509fbaa62645c58b7ab1a432eb8727 P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Becoming Leader. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Running, Role: LEADER
I20240503 02:13:23.936971 4240 ts_tablet_manager.cc:1398] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:13:23.937459 4177 consensus_queue.cc:237] T 88509fbaa62645c58b7ab1a432eb8727 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:23.940604 29895 tablet_copy_service.cc:342] P 16440e365430405fb10366ab70e2b429: Request end of tablet copy session 1f10f80f13c84e80b03eda7085a3b012-58e1aaecf47944569296fc1e0780efb5 received from {username='slave'} at 127.22.108.4:51273
I20240503 02:13:23.940761 29895 tablet_copy_service.cc:433] P 16440e365430405fb10366ab70e2b429: ending tablet copy session 1f10f80f13c84e80b03eda7085a3b012-58e1aaecf47944569296fc1e0780efb5 on tablet 58e1aaecf47944569296fc1e0780efb5 with peer 1f10f80f13c84e80b03eda7085a3b012
I20240503 02:13:23.942035 3795 leader_election.cc:288] T 88509fbaa62645c58b7ab1a432eb8727 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Requested vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:13:23.944881 30132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88509fbaa62645c58b7ab1a432eb8727" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:13:23.947247 32708 leader_election.cc:302] T 88509fbaa62645c58b7ab1a432eb8727 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead; no voters: 16440e365430405fb10366ab70e2b429, a9094240860e43bfb129193f3bfd5be6
I20240503 02:13:23.949544 3795 raft_consensus.cc:2726] T 88509fbaa62645c58b7ab1a432eb8727 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 02:13:23.954512 4073 tablet_bootstrap.cc:492] T acb04344073b4ceaaa6d404c1a774434 P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:23.954702 4073 ts_tablet_manager.cc:1367] T acb04344073b4ceaaa6d404c1a774434 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.048s user 0.001s sys 0.010s
I20240503 02:13:23.955462 4073 raft_consensus.cc:348] T acb04344073b4ceaaa6d404c1a774434 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:23.955701 4073 raft_consensus.cc:374] T acb04344073b4ceaaa6d404c1a774434 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:23.955766 4073 raft_consensus.cc:729] T acb04344073b4ceaaa6d404c1a774434 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:23.955991 4073 consensus_queue.cc:260] T acb04344073b4ceaaa6d404c1a774434 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:23.956720 4073 ts_tablet_manager.cc:1398] T acb04344073b4ceaaa6d404c1a774434 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:13:23.959288 29729 catalog_manager.cc:5484] T 88509fbaa62645c58b7ab1a432eb8727 P a9094240860e43bfb129193f3bfd5be6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9094240860e43bfb129193f3bfd5be6 (127.22.108.3). New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:23.964131 32640 raft_consensus.cc:1203] T 493c813acc27492fb952791cf2dc12a2 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:23.973986 4217 tablet_bootstrap.cc:492] T acb04344073b4ceaaa6d404c1a774434 P 9a4d92bc25374fb3960fde7d10be309d: No bootstrap required, opened a new log
I20240503 02:13:23.974203 4217 ts_tablet_manager.cc:1367] T acb04344073b4ceaaa6d404c1a774434 P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.059s user 0.001s sys 0.008s
I20240503 02:13:23.975142 4217 raft_consensus.cc:348] T acb04344073b4ceaaa6d404c1a774434 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:23.975446 4217 raft_consensus.cc:374] T acb04344073b4ceaaa6d404c1a774434 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:23.975529 4217 raft_consensus.cc:729] T acb04344073b4ceaaa6d404c1a774434 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:23.975888 4217 consensus_queue.cc:260] T acb04344073b4ceaaa6d404c1a774434 P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:23.976630 4217 ts_tablet_manager.cc:1398] T acb04344073b4ceaaa6d404c1a774434 P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:13:23.985167 4261 ts_tablet_manager.cc:1888] T f203eb4a88544e9aa778694964c51522 P a9094240860e43bfb129193f3bfd5be6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:23.985311 4261 log.cc:1196] T f203eb4a88544e9aa778694964c51522 P a9094240860e43bfb129193f3bfd5be6: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/wals/f203eb4a88544e9aa778694964c51522
I20240503 02:13:23.985831 4261 ts_tablet_manager.cc:1909] T f203eb4a88544e9aa778694964c51522 P a9094240860e43bfb129193f3bfd5be6: Deleting consensus metadata
I20240503 02:13:23.987411 29718 catalog_manager.cc:4830] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): tablet f203eb4a88544e9aa778694964c51522 (table rebalancer_dynamic_table_new_name [id=b9cf1367dc464a5692bd5b989bcc627e]) successfully deleted
I20240503 02:13:23.989105 4260 ts_tablet_manager.cc:1888] T f203eb4a88544e9aa778694964c51522 P f575ce92c0224eeb80b9fae5f6561fbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:23.989255 4260 log.cc:1196] T f203eb4a88544e9aa778694964c51522 P f575ce92c0224eeb80b9fae5f6561fbe: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal/wals/f203eb4a88544e9aa778694964c51522
I20240503 02:13:23.989763 4260 ts_tablet_manager.cc:1909] T f203eb4a88544e9aa778694964c51522 P f575ce92c0224eeb80b9fae5f6561fbe: Deleting consensus metadata
I20240503 02:13:23.990844 29718 catalog_manager.cc:4830] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): tablet f203eb4a88544e9aa778694964c51522 (table rebalancer_dynamic_table_new_name [id=b9cf1367dc464a5692bd5b989bcc627e]) successfully deleted
I20240503 02:13:23.992982 3795 raft_consensus.cc:482] T acb04344073b4ceaaa6d404c1a774434 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:23.993184 3795 raft_consensus.cc:504] T acb04344073b4ceaaa6d404c1a774434 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:23.994112 3795 leader_election.cc:288] T acb04344073b4ceaaa6d404c1a774434 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:13:23.994714 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb04344073b4ceaaa6d404c1a774434" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
W20240503 02:13:23.995281 32709 leader_election.cc:341] T acb04344073b4ceaaa6d404c1a774434 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:23.996343 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb04344073b4ceaaa6d404c1a774434" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d" is_pre_election: true
I20240503 02:13:23.996569 32640 raft_consensus.cc:2454] T acb04344073b4ceaaa6d404c1a774434 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:13:23.997443 32709 leader_election.cc:302] T acb04344073b4ceaaa6d404c1a774434 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, 9a4d92bc25374fb3960fde7d10be309d; no voters: 155735c7f8b14997ac4c770aee2a9401
I20240503 02:13:23.997822 3795 raft_consensus.cc:2781] T acb04344073b4ceaaa6d404c1a774434 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:23.997921 3795 raft_consensus.cc:482] T acb04344073b4ceaaa6d404c1a774434 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:23.998009 3795 raft_consensus.cc:3037] T acb04344073b4ceaaa6d404c1a774434 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:24.000880 3795 raft_consensus.cc:504] T acb04344073b4ceaaa6d404c1a774434 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:24.004824 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb04344073b4ceaaa6d404c1a774434" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
W20240503 02:13:24.005443 32709 leader_election.cc:341] T acb04344073b4ceaaa6d404c1a774434 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:24.005842 3795 leader_election.cc:288] T acb04344073b4ceaaa6d404c1a774434 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Requested vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:13:24.008121 4259 ts_tablet_manager.cc:1888] T f203eb4a88544e9aa778694964c51522 P 8a666e798fe8477d8dd1e51b385d5ead: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:24.009874 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb04344073b4ceaaa6d404c1a774434" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a4d92bc25374fb3960fde7d10be309d"
I20240503 02:13:24.010088 32640 raft_consensus.cc:3037] T acb04344073b4ceaaa6d404c1a774434 P 9a4d92bc25374fb3960fde7d10be309d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:24.012750 4265 tablet_bootstrap.cc:492] T acb04344073b4ceaaa6d404c1a774434 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:13:24.012940 32640 raft_consensus.cc:2454] T acb04344073b4ceaaa6d404c1a774434 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:13:24.015465 4259 log.cc:1196] T f203eb4a88544e9aa778694964c51522 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal/wals/f203eb4a88544e9aa778694964c51522
I20240503 02:13:24.018527 32709 leader_election.cc:302] T acb04344073b4ceaaa6d404c1a774434 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, 9a4d92bc25374fb3960fde7d10be309d; no voters: 155735c7f8b14997ac4c770aee2a9401
I20240503 02:13:24.019362 3795 raft_consensus.cc:2781] T acb04344073b4ceaaa6d404c1a774434 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:24.019516 3795 raft_consensus.cc:686] T acb04344073b4ceaaa6d404c1a774434 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Becoming Leader. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Running, Role: LEADER
I20240503 02:13:24.021508 3795 consensus_queue.cc:237] T acb04344073b4ceaaa6d404c1a774434 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:24.025331 4259 ts_tablet_manager.cc:1909] T f203eb4a88544e9aa778694964c51522 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting consensus metadata
I20240503 02:13:24.026407 29719 catalog_manager.cc:4830] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): tablet f203eb4a88544e9aa778694964c51522 (table rebalancer_dynamic_table_new_name [id=b9cf1367dc464a5692bd5b989bcc627e]) successfully deleted
I20240503 02:13:24.029640 4265 tablet_bootstrap.cc:654] T acb04344073b4ceaaa6d404c1a774434 P 155735c7f8b14997ac4c770aee2a9401: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:24.045958 29732 catalog_manager.cc:5484] T acb04344073b4ceaaa6d404c1a774434 P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6). New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:24.057776 4263 tablet_copy_client.cc:308] T e4fe8ae88ae843609a66c7a2e6c7b4af P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.3:42679
I20240503 02:13:24.060503 30156 tablet_copy_service.cc:140] P a9094240860e43bfb129193f3bfd5be6: Received BeginTabletCopySession request for tablet e4fe8ae88ae843609a66c7a2e6c7b4af from peer 155735c7f8b14997ac4c770aee2a9401 ({username='slave'} at 127.22.108.2:56649)
I20240503 02:13:24.060689 30156 tablet_copy_service.cc:161] P a9094240860e43bfb129193f3bfd5be6: Beginning new tablet copy session on tablet e4fe8ae88ae843609a66c7a2e6c7b4af from peer 155735c7f8b14997ac4c770aee2a9401 at {username='slave'} at 127.22.108.2:56649: session id = 155735c7f8b14997ac4c770aee2a9401-e4fe8ae88ae843609a66c7a2e6c7b4af
I20240503 02:13:24.063634 30156 tablet_copy_source_session.cc:205] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:24.070477 4265 tablet_bootstrap.cc:492] T acb04344073b4ceaaa6d404c1a774434 P 155735c7f8b14997ac4c770aee2a9401: No bootstrap required, opened a new log
I20240503 02:13:24.070758 4265 ts_tablet_manager.cc:1367] T acb04344073b4ceaaa6d404c1a774434 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.058s user 0.007s sys 0.003s
I20240503 02:13:24.071731 4265 raft_consensus.cc:348] T acb04344073b4ceaaa6d404c1a774434 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:24.071949 4265 raft_consensus.cc:374] T acb04344073b4ceaaa6d404c1a774434 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:24.072023 4265 raft_consensus.cc:729] T acb04344073b4ceaaa6d404c1a774434 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:13:24.072275 4265 consensus_queue.cc:260] T acb04344073b4ceaaa6d404c1a774434 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:24.072888 4263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4fe8ae88ae843609a66c7a2e6c7b4af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:24.073462 4265 ts_tablet_manager.cc:1398] T acb04344073b4ceaaa6d404c1a774434 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:13:24.094331 4273 consensus_queue.cc:1035] T 9cd1418eb38d420c9c41d660476eafb0 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:24.113986 4263 tablet_copy_client.cc:783] T e4fe8ae88ae843609a66c7a2e6c7b4af P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:24.114459 4263 tablet_copy_client.cc:647] T e4fe8ae88ae843609a66c7a2e6c7b4af P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:24.117870 4263 tablet_copy_client.cc:523] T e4fe8ae88ae843609a66c7a2e6c7b4af P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:24.122123 4263 tablet_bootstrap.cc:492] T e4fe8ae88ae843609a66c7a2e6c7b4af P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:13:24.136360 3956 consensus_queue.cc:1035] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:24.137701 4263 tablet_bootstrap.cc:492] T e4fe8ae88ae843609a66c7a2e6c7b4af P 155735c7f8b14997ac4c770aee2a9401: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:24.138185 4263 tablet_bootstrap.cc:492] T e4fe8ae88ae843609a66c7a2e6c7b4af P 155735c7f8b14997ac4c770aee2a9401: Bootstrap complete.
I20240503 02:13:24.138490 4263 ts_tablet_manager.cc:1367] T e4fe8ae88ae843609a66c7a2e6c7b4af P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.016s user 0.014s sys 0.000s
I20240503 02:13:24.139271 4263 raft_consensus.cc:348] T e4fe8ae88ae843609a66c7a2e6c7b4af P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:24.139619 4263 raft_consensus.cc:729] T e4fe8ae88ae843609a66c7a2e6c7b4af P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:13:24.142881 4263 consensus_queue.cc:260] T e4fe8ae88ae843609a66c7a2e6c7b4af P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:24.143748 4263 ts_tablet_manager.cc:1398] T e4fe8ae88ae843609a66c7a2e6c7b4af P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 02:13:24.147390 30265 raft_consensus.cc:1203] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:24.147807 30156 tablet_copy_service.cc:342] P a9094240860e43bfb129193f3bfd5be6: Request end of tablet copy session 155735c7f8b14997ac4c770aee2a9401-e4fe8ae88ae843609a66c7a2e6c7b4af received from {username='slave'} at 127.22.108.2:56649
I20240503 02:13:24.147910 30156 tablet_copy_service.cc:433] P a9094240860e43bfb129193f3bfd5be6: ending tablet copy session 155735c7f8b14997ac4c770aee2a9401-e4fe8ae88ae843609a66c7a2e6c7b4af on tablet e4fe8ae88ae843609a66c7a2e6c7b4af with peer 155735c7f8b14997ac4c770aee2a9401
I20240503 02:13:24.163695 3956 consensus_queue.cc:1035] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:24.176849 29729 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:13:24.205770 30005 raft_consensus.cc:3037] T acb04344073b4ceaaa6d404c1a774434 P 155735c7f8b14997ac4c770aee2a9401 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:24.206339 4147 consensus_queue.cc:1035] T 9cd1418eb38d420c9c41d660476eafb0 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:24.208752 30005 raft_consensus.cc:1261] T acb04344073b4ceaaa6d404c1a774434 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:24.205838 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:24.212600 3821 consensus_queue.cc:1035] T acb04344073b4ceaaa6d404c1a774434 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:24.215883 32640 raft_consensus.cc:1261] T acb04344073b4ceaaa6d404c1a774434 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:24.216675 3821 consensus_queue.cc:1035] T acb04344073b4ceaaa6d404c1a774434 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:24.244151 29728 catalog_manager.cc:5576] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873) does not have the latest schema for tablet acb04344073b4ceaaa6d404c1a774434 (table rebalancer_dynamic_table [id=d5adfab772b6453b980eeb511818c012]). Expected version 1 got 0
I20240503 02:13:24.251657 29729 catalog_manager.cc:5576] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595) does not have the latest schema for tablet acb04344073b4ceaaa6d404c1a774434 (table rebalancer_dynamic_table [id=d5adfab772b6453b980eeb511818c012]). Expected version 1 got 0
I20240503 02:13:24.258183 4307 tablet.cc:1722] T acb04344073b4ceaaa6d404c1a774434 P 9a4d92bc25374fb3960fde7d10be309d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:24.258229 4306 tablet.cc:1722] T acb04344073b4ceaaa6d404c1a774434 P 155735c7f8b14997ac4c770aee2a9401: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:24.258795 4307 tablet.cc:1620] T acb04344073b4ceaaa6d404c1a774434 P 9a4d92bc25374fb3960fde7d10be309d: MemRowSet was empty: no flush needed.
I20240503 02:13:24.258822 4306 tablet.cc:1620] T acb04344073b4ceaaa6d404c1a774434 P 155735c7f8b14997ac4c770aee2a9401: MemRowSet was empty: no flush needed.
I20240503 02:13:24.259490 4305 tablet.cc:1722] T acb04344073b4ceaaa6d404c1a774434 P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:24.259987 4305 tablet.cc:1620] T acb04344073b4ceaaa6d404c1a774434 P 8a666e798fe8477d8dd1e51b385d5ead: MemRowSet was empty: no flush needed.
I20240503 02:13:24.261291 29732 catalog_manager.cc:5576] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595) does not have the latest schema for tablet acb04344073b4ceaaa6d404c1a774434 (table rebalancer_dynamic_table [id=d5adfab772b6453b980eeb511818c012]). Expected version 1 got 0
I20240503 02:13:24.263381 29729 catalog_manager.cc:5576] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873) does not have the latest schema for tablet acb04344073b4ceaaa6d404c1a774434 (table rebalancer_dynamic_table [id=d5adfab772b6453b980eeb511818c012]). Expected version 1 got 0
I20240503 02:13:24.273058 30244 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:24.276770 29728 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=d5adfab772b6453b980eeb511818c012] alter complete (version 1)
I20240503 02:13:24.286130 29853 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:24.353312 32619 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:24.355476 4177 consensus_queue.cc:1035] T 88509fbaa62645c58b7ab1a432eb8727 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:24.379873 30114 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:24.383037 32754 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:24.387197 29728 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:13:24.397375 4305 tablet.cc:1722] T acb04344073b4ceaaa6d404c1a774434 P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:24.397907 4305 tablet.cc:1620] T acb04344073b4ceaaa6d404c1a774434 P 8a666e798fe8477d8dd1e51b385d5ead: MemRowSet was empty: no flush needed.
I20240503 02:13:24.399741 4307 tablet.cc:1722] T acb04344073b4ceaaa6d404c1a774434 P 9a4d92bc25374fb3960fde7d10be309d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:24.400235 4307 tablet.cc:1620] T acb04344073b4ceaaa6d404c1a774434 P 9a4d92bc25374fb3960fde7d10be309d: MemRowSet was empty: no flush needed.
I20240503 02:13:24.401675 4306 tablet.cc:1722] T acb04344073b4ceaaa6d404c1a774434 P 155735c7f8b14997ac4c770aee2a9401: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:24.402148 4306 tablet.cc:1620] T acb04344073b4ceaaa6d404c1a774434 P 155735c7f8b14997ac4c770aee2a9401: MemRowSet was empty: no flush needed.
I20240503 02:13:24.410710 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:24.420217 29729 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table [id=d5adfab772b6453b980eeb511818c012] alter complete (version 2)
I20240503 02:13:24.426139 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:24.430440 30004 raft_consensus.cc:1203] T e4fe8ae88ae843609a66c7a2e6c7b4af P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:24.464769 4177 consensus_queue.cc:1035] T 88509fbaa62645c58b7ab1a432eb8727 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:24.524013 29732 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:13:24.536762 4305 tablet.cc:1722] T acb04344073b4ceaaa6d404c1a774434 P 8a666e798fe8477d8dd1e51b385d5ead: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:24.541455 4307 tablet.cc:1722] T acb04344073b4ceaaa6d404c1a774434 P 9a4d92bc25374fb3960fde7d10be309d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:24.542975 4306 tablet.cc:1722] T acb04344073b4ceaaa6d404c1a774434 P 155735c7f8b14997ac4c770aee2a9401: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:24.552958 29732 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: rebalancer_dynamic_table_new_name [id=d5adfab772b6453b980eeb511818c012] alter complete (version 3)
I20240503 02:13:24.560837 29728 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_extra_table_54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:24.575054 29853 tablet_service.cc:1467] Processing CreateTablet for tablet 3533364f91c3433a8c1536b3eaee7424 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=078d9afd4235410089ac085e2345886c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:24.575495 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3533364f91c3433a8c1536b3eaee7424. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:24.575387 547 tablet_service.cc:1467] Processing CreateTablet for tablet 3533364f91c3433a8c1536b3eaee7424 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=078d9afd4235410089ac085e2345886c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:24.575423 30244 tablet_service.cc:1467] Processing CreateTablet for tablet 3533364f91c3433a8c1536b3eaee7424 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=078d9afd4235410089ac085e2345886c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:24.575781 547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3533364f91c3433a8c1536b3eaee7424. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:24.575838 30244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3533364f91c3433a8c1536b3eaee7424. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:24.583493 30114 tablet_service.cc:1467] Processing CreateTablet for tablet ebcd2c6e1722442fbe603cfcccb56ab0 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=078d9afd4235410089ac085e2345886c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:24.583917 30114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebcd2c6e1722442fbe603cfcccb56ab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:24.589188 32754 tablet_service.cc:1467] Processing CreateTablet for tablet ebcd2c6e1722442fbe603cfcccb56ab0 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=078d9afd4235410089ac085e2345886c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:24.589655 32754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebcd2c6e1722442fbe603cfcccb56ab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:24.595134 415 tablet_service.cc:1467] Processing CreateTablet for tablet ebcd2c6e1722442fbe603cfcccb56ab0 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=078d9afd4235410089ac085e2345886c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:24.595733 415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebcd2c6e1722442fbe603cfcccb56ab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:24.601015 4325 tablet_bootstrap.cc:492] T 3533364f91c3433a8c1536b3eaee7424 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:24.611991 30115 tablet_service.cc:1467] Processing CreateTablet for tablet b90b887127f44c0d920225be10077c64 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=078d9afd4235410089ac085e2345886c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:24.612552 30115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b90b887127f44c0d920225be10077c64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:24.616078 4325 tablet_bootstrap.cc:654] T 3533364f91c3433a8c1536b3eaee7424 P f575ce92c0224eeb80b9fae5f6561fbe: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:24.620949 4326 tablet_bootstrap.cc:492] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:13:24.626801 416 tablet_service.cc:1467] Processing CreateTablet for tablet b90b887127f44c0d920225be10077c64 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=078d9afd4235410089ac085e2345886c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:24.627233 416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b90b887127f44c0d920225be10077c64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:24.627444 30245 tablet_service.cc:1467] Processing CreateTablet for tablet b90b887127f44c0d920225be10077c64 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=078d9afd4235410089ac085e2345886c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:24.627878 30245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b90b887127f44c0d920225be10077c64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:24.636715 4326 tablet_bootstrap.cc:654] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:24.656057 4326 tablet_bootstrap.cc:492] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429: No bootstrap required, opened a new log
I20240503 02:13:24.656296 4326 ts_tablet_manager.cc:1367] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.035s user 0.005s sys 0.000s
I20240503 02:13:24.657078 4326 raft_consensus.cc:348] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:24.657352 4326 raft_consensus.cc:374] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:24.657451 4326 raft_consensus.cc:729] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:13:24.657729 4326 consensus_queue.cc:260] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:24.661665 29729 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:24.661859 29729 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37220:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:24.662576 4326 ts_tablet_manager.cc:1398] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 02:13:24.663638 4330 tablet_bootstrap.cc:492] T ebcd2c6e1722442fbe603cfcccb56ab0 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:24.665015 4329 tablet_bootstrap.cc:492] T 3533364f91c3433a8c1536b3eaee7424 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:13:24.667047 29729 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e72f3141a78a4d0e811377c9c1f11f00: Sending DeleteTablet for 3 replicas of tablet acb04344073b4ceaaa6d404c1a774434
I20240503 02:13:24.668471 29985 tablet_service.cc:1514] Processing DeleteTablet for tablet acb04344073b4ceaaa6d404c1a774434 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:24 UTC) from {username='slave'} at 127.0.0.1:52158
I20240503 02:13:24.668956 32755 tablet_service.cc:1514] Processing DeleteTablet for tablet acb04344073b4ceaaa6d404c1a774434 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:24 UTC) from {username='slave'} at 127.0.0.1:46360
I20240503 02:13:24.669824 4330 tablet_bootstrap.cc:654] T ebcd2c6e1722442fbe603cfcccb56ab0 P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:24.670145 32619 tablet_service.cc:1514] Processing DeleteTablet for tablet acb04344073b4ceaaa6d404c1a774434 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:24 UTC) from {username='slave'} at 127.0.0.1:38200
I20240503 02:13:24.672917 4328 tablet_bootstrap.cc:492] T ebcd2c6e1722442fbe603cfcccb56ab0 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:13:24.676533 4328 tablet_bootstrap.cc:654] T ebcd2c6e1722442fbe603cfcccb56ab0 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:24.680332 4325 tablet_bootstrap.cc:492] T 3533364f91c3433a8c1536b3eaee7424 P f575ce92c0224eeb80b9fae5f6561fbe: No bootstrap required, opened a new log
I20240503 02:13:24.680549 4325 ts_tablet_manager.cc:1367] T 3533364f91c3433a8c1536b3eaee7424 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.080s user 0.007s sys 0.014s
I20240503 02:13:24.681362 4325 raft_consensus.cc:348] T 3533364f91c3433a8c1536b3eaee7424 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:24.681635 4325 raft_consensus.cc:374] T 3533364f91c3433a8c1536b3eaee7424 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:24.681751 4325 raft_consensus.cc:729] T 3533364f91c3433a8c1536b3eaee7424 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:24.682047 4325 consensus_queue.cc:260] T 3533364f91c3433a8c1536b3eaee7424 P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:24.682632 4325 ts_tablet_manager.cc:1398] T 3533364f91c3433a8c1536b3eaee7424 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:13:24.686394 4334 tablet_replica.cc:330] T acb04344073b4ceaaa6d404c1a774434 P 8a666e798fe8477d8dd1e51b385d5ead: stopping tablet replica
I20240503 02:13:24.686669 4334 raft_consensus.cc:2229] T acb04344073b4ceaaa6d404c1a774434 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:24.687095 4334 raft_consensus.cc:2258] T acb04344073b4ceaaa6d404c1a774434 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:24.687948 4334 ts_tablet_manager.cc:1875] T acb04344073b4ceaaa6d404c1a774434 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:24.693363 4331 tablet_bootstrap.cc:492] T ebcd2c6e1722442fbe603cfcccb56ab0 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:24.695088 4331 tablet_bootstrap.cc:654] T ebcd2c6e1722442fbe603cfcccb56ab0 P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:24.701196 4331 tablet_bootstrap.cc:492] T ebcd2c6e1722442fbe603cfcccb56ab0 P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:24.701452 4331 ts_tablet_manager.cc:1367] T ebcd2c6e1722442fbe603cfcccb56ab0 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240503 02:13:24.703452 4331 raft_consensus.cc:348] T ebcd2c6e1722442fbe603cfcccb56ab0 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:24.705546 4331 raft_consensus.cc:374] T ebcd2c6e1722442fbe603cfcccb56ab0 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:24.706233 4331 raft_consensus.cc:729] T ebcd2c6e1722442fbe603cfcccb56ab0 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:24.706507 4331 consensus_queue.cc:260] T ebcd2c6e1722442fbe603cfcccb56ab0 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:24.707171 4331 ts_tablet_manager.cc:1398] T ebcd2c6e1722442fbe603cfcccb56ab0 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:13:24.706094 4334 ts_tablet_manager.cc:1888] T acb04344073b4ceaaa6d404c1a774434 P 8a666e798fe8477d8dd1e51b385d5ead: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:24.708143 4334 log.cc:1196] T acb04344073b4ceaaa6d404c1a774434 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal/wals/acb04344073b4ceaaa6d404c1a774434
I20240503 02:13:24.708777 4334 ts_tablet_manager.cc:1909] T acb04344073b4ceaaa6d404c1a774434 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting consensus metadata
I20240503 02:13:24.715162 29719 catalog_manager.cc:4830] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): tablet acb04344073b4ceaaa6d404c1a774434 (table rebalancer_dynamic_table_new_name [id=d5adfab772b6453b980eeb511818c012]) successfully deleted
I20240503 02:13:24.729354 4333 tablet_replica.cc:330] T acb04344073b4ceaaa6d404c1a774434 P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:13:24.729729 4333 raft_consensus.cc:2229] T acb04344073b4ceaaa6d404c1a774434 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:24.730036 4333 raft_consensus.cc:2258] T acb04344073b4ceaaa6d404c1a774434 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:24.731550 4333 ts_tablet_manager.cc:1875] T acb04344073b4ceaaa6d404c1a774434 P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:24.733547 4335 tablet_replica.cc:330] T acb04344073b4ceaaa6d404c1a774434 P 9a4d92bc25374fb3960fde7d10be309d: stopping tablet replica
I20240503 02:13:24.734159 4335 raft_consensus.cc:2229] T acb04344073b4ceaaa6d404c1a774434 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:24.734575 4335 raft_consensus.cc:2258] T acb04344073b4ceaaa6d404c1a774434 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:24.735805 4335 ts_tablet_manager.cc:1875] T acb04344073b4ceaaa6d404c1a774434 P 9a4d92bc25374fb3960fde7d10be309d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:24.744765 4328 tablet_bootstrap.cc:492] T ebcd2c6e1722442fbe603cfcccb56ab0 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:13:24.745051 4328 ts_tablet_manager.cc:1367] T ebcd2c6e1722442fbe603cfcccb56ab0 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.072s user 0.018s sys 0.016s
I20240503 02:13:24.746027 4328 raft_consensus.cc:348] T ebcd2c6e1722442fbe603cfcccb56ab0 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:24.746349 4328 raft_consensus.cc:374] T ebcd2c6e1722442fbe603cfcccb56ab0 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:24.746428 4328 raft_consensus.cc:729] T ebcd2c6e1722442fbe603cfcccb56ab0 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:13:24.746690 4328 consensus_queue.cc:260] T ebcd2c6e1722442fbe603cfcccb56ab0 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:24.747350 4328 ts_tablet_manager.cc:1398] T ebcd2c6e1722442fbe603cfcccb56ab0 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:13:24.747686 4328 tablet_bootstrap.cc:492] T b90b887127f44c0d920225be10077c64 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:13:24.750293 4329 tablet_bootstrap.cc:654] T 3533364f91c3433a8c1536b3eaee7424 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:24.751806 4330 tablet_bootstrap.cc:492] T ebcd2c6e1722442fbe603cfcccb56ab0 P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:24.751991 4330 ts_tablet_manager.cc:1367] T ebcd2c6e1722442fbe603cfcccb56ab0 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.088s user 0.009s sys 0.032s
I20240503 02:13:24.752982 4330 raft_consensus.cc:348] T ebcd2c6e1722442fbe603cfcccb56ab0 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:24.753291 4330 raft_consensus.cc:374] T ebcd2c6e1722442fbe603cfcccb56ab0 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:24.753417 4330 raft_consensus.cc:729] T ebcd2c6e1722442fbe603cfcccb56ab0 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:24.753717 4330 consensus_queue.cc:260] T ebcd2c6e1722442fbe603cfcccb56ab0 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:24.754437 4330 ts_tablet_manager.cc:1398] T ebcd2c6e1722442fbe603cfcccb56ab0 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:24.754757 4330 tablet_bootstrap.cc:492] T b90b887127f44c0d920225be10077c64 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:24.761691 4330 tablet_bootstrap.cc:654] T b90b887127f44c0d920225be10077c64 P fbbc2be6382e47908ecb7c68c367bd26: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:24.769181 4329 tablet_bootstrap.cc:492] T 3533364f91c3433a8c1536b3eaee7424 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:13:24.769346 4329 ts_tablet_manager.cc:1367] T 3533364f91c3433a8c1536b3eaee7424 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.104s user 0.005s sys 0.002s
I20240503 02:13:24.770067 4329 raft_consensus.cc:348] T 3533364f91c3433a8c1536b3eaee7424 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:24.770227 4328 tablet_bootstrap.cc:654] T b90b887127f44c0d920225be10077c64 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:24.770272 4329 raft_consensus.cc:374] T 3533364f91c3433a8c1536b3eaee7424 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:24.770354 4329 raft_consensus.cc:729] T 3533364f91c3433a8c1536b3eaee7424 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:13:24.770596 4329 consensus_queue.cc:260] T 3533364f91c3433a8c1536b3eaee7424 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:24.774756 4329 ts_tablet_manager.cc:1398] T 3533364f91c3433a8c1536b3eaee7424 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:13:24.775118 4329 tablet_bootstrap.cc:492] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:13:24.775221 4333 ts_tablet_manager.cc:1888] T acb04344073b4ceaaa6d404c1a774434 P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:24.775406 4333 log.cc:1196] T acb04344073b4ceaaa6d404c1a774434 P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/acb04344073b4ceaaa6d404c1a774434
I20240503 02:13:24.775993 4333 ts_tablet_manager.cc:1909] T acb04344073b4ceaaa6d404c1a774434 P 155735c7f8b14997ac4c770aee2a9401: Deleting consensus metadata
I20240503 02:13:24.776253 4335 ts_tablet_manager.cc:1888] T acb04344073b4ceaaa6d404c1a774434 P 9a4d92bc25374fb3960fde7d10be309d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:24.776369 4330 tablet_bootstrap.cc:492] T b90b887127f44c0d920225be10077c64 P fbbc2be6382e47908ecb7c68c367bd26: No bootstrap required, opened a new log
I20240503 02:13:24.776672 4329 tablet_bootstrap.cc:654] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:24.776687 4335 log.cc:1196] T acb04344073b4ceaaa6d404c1a774434 P 9a4d92bc25374fb3960fde7d10be309d: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal/wals/acb04344073b4ceaaa6d404c1a774434
I20240503 02:13:24.777089 4330 ts_tablet_manager.cc:1367] T b90b887127f44c0d920225be10077c64 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.002s
I20240503 02:13:24.777237 4335 ts_tablet_manager.cc:1909] T acb04344073b4ceaaa6d404c1a774434 P 9a4d92bc25374fb3960fde7d10be309d: Deleting consensus metadata
I20240503 02:13:24.778214 4330 raft_consensus.cc:348] T b90b887127f44c0d920225be10077c64 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:24.779882 29719 catalog_manager.cc:4830] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): tablet acb04344073b4ceaaa6d404c1a774434 (table rebalancer_dynamic_table_new_name [id=d5adfab772b6453b980eeb511818c012]) successfully deleted
I20240503 02:13:24.780246 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet acb04344073b4ceaaa6d404c1a774434 (table rebalancer_dynamic_table_new_name [id=d5adfab772b6453b980eeb511818c012]) successfully deleted
I20240503 02:13:24.780742 4330 raft_consensus.cc:374] T b90b887127f44c0d920225be10077c64 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:24.781049 4330 raft_consensus.cc:729] T b90b887127f44c0d920225be10077c64 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:24.781250 4328 tablet_bootstrap.cc:492] T b90b887127f44c0d920225be10077c64 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:13:24.781436 4328 ts_tablet_manager.cc:1367] T b90b887127f44c0d920225be10077c64 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.034s user 0.001s sys 0.004s
I20240503 02:13:24.781724 4330 consensus_queue.cc:260] T b90b887127f44c0d920225be10077c64 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:24.782337 4328 raft_consensus.cc:348] T b90b887127f44c0d920225be10077c64 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:24.782596 4328 raft_consensus.cc:374] T b90b887127f44c0d920225be10077c64 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:24.782696 4328 raft_consensus.cc:729] T b90b887127f44c0d920225be10077c64 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:13:24.782927 4330 ts_tablet_manager.cc:1398] T b90b887127f44c0d920225be10077c64 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:13:24.782955 4328 consensus_queue.cc:260] T b90b887127f44c0d920225be10077c64 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:24.783815 4328 ts_tablet_manager.cc:1398] T b90b887127f44c0d920225be10077c64 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:13:24.788514 4329 tablet_bootstrap.cc:492] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:13:24.788801 4329 ts_tablet_manager.cc:1367] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.001s
I20240503 02:13:24.789741 4329 raft_consensus.cc:348] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:24.790112 4329 raft_consensus.cc:374] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:24.790228 4329 raft_consensus.cc:729] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:13:24.790553 4329 consensus_queue.cc:260] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:24.791473 4329 ts_tablet_manager.cc:1398] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:24.799696 29728 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37220:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:13:24.811791 30244 tablet_service.cc:1467] Processing CreateTablet for tablet d2a8d96585fd4b4881d1bd028b1a9d36 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c7b2e6458bdf4982bbf7b831ad18f4a3]), partition=
I20240503 02:13:24.811831 32755 tablet_service.cc:1467] Processing CreateTablet for tablet d2a8d96585fd4b4881d1bd028b1a9d36 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c7b2e6458bdf4982bbf7b831ad18f4a3]), partition=
I20240503 02:13:24.812186 30244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2a8d96585fd4b4881d1bd028b1a9d36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:24.812223 32755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2a8d96585fd4b4881d1bd028b1a9d36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:24.817246 4331 tablet_bootstrap.cc:492] T d2a8d96585fd4b4881d1bd028b1a9d36 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:24.818902 4331 tablet_bootstrap.cc:654] T d2a8d96585fd4b4881d1bd028b1a9d36 P 8a666e798fe8477d8dd1e51b385d5ead: Neither blocks nor log segments found. Creating new log.
W20240503 02:13:24.825809 30200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.7:44241, user_credentials={real_user=slave}} blocked reactor thread for 69390us
I20240503 02:13:24.830108 30114 tablet_service.cc:1467] Processing CreateTablet for tablet d2a8d96585fd4b4881d1bd028b1a9d36 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c7b2e6458bdf4982bbf7b831ad18f4a3]), partition=
I20240503 02:13:24.830569 30114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2a8d96585fd4b4881d1bd028b1a9d36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:24.841575 4091 raft_consensus.cc:482] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:24.841735 4091 raft_consensus.cc:504] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:24.842453 4091 leader_election.cc:288] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:24.842813 30132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b90b887127f44c0d920225be10077c64" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:13:24.843010 30132 raft_consensus.cc:2454] T b90b887127f44c0d920225be10077c64 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:13:24.843430 30199 leader_election.cc:302] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:13:24.843708 4091 raft_consensus.cc:2781] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:24.843816 4091 raft_consensus.cc:482] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:24.843883 4091 raft_consensus.cc:3037] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:24.844974 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b90b887127f44c0d920225be10077c64" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:13:24.845247 436 raft_consensus.cc:2454] T b90b887127f44c0d920225be10077c64 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 0.
I20240503 02:13:24.846737 4091 raft_consensus.cc:504] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:24.847905 30132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b90b887127f44c0d920225be10077c64" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:13:24.848150 30132 raft_consensus.cc:3037] T b90b887127f44c0d920225be10077c64 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:24.847463 4091 leader_election.cc:288] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Requested vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:24.849889 4329 tablet_bootstrap.cc:492] T d2a8d96585fd4b4881d1bd028b1a9d36 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:13:24.851281 4329 tablet_bootstrap.cc:654] T d2a8d96585fd4b4881d1bd028b1a9d36 P 1f10f80f13c84e80b03eda7085a3b012: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:24.853082 30132 raft_consensus.cc:2454] T b90b887127f44c0d920225be10077c64 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:13:24.853518 30199 leader_election.cc:302] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:13:24.853837 4091 raft_consensus.cc:2781] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:24.853945 4091 raft_consensus.cc:686] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Becoming Leader. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Running, Role: LEADER
I20240503 02:13:24.854161 4091 consensus_queue.cc:237] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:24.855517 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b90b887127f44c0d920225be10077c64" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:13:24.855736 436 raft_consensus.cc:3037] T b90b887127f44c0d920225be10077c64 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:24.858605 436 raft_consensus.cc:2454] T b90b887127f44c0d920225be10077c64 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
I20240503 02:13:24.863459 29729 catalog_manager.cc:5484] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4). New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:24.867662 4328 tablet_bootstrap.cc:492] T d2a8d96585fd4b4881d1bd028b1a9d36 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:13:24.869287 4328 tablet_bootstrap.cc:654] T d2a8d96585fd4b4881d1bd028b1a9d36 P a9094240860e43bfb129193f3bfd5be6: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:24.879791 4328 tablet_bootstrap.cc:492] T d2a8d96585fd4b4881d1bd028b1a9d36 P a9094240860e43bfb129193f3bfd5be6: No bootstrap required, opened a new log
I20240503 02:13:24.879945 4328 ts_tablet_manager.cc:1367] T d2a8d96585fd4b4881d1bd028b1a9d36 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.001s
I20240503 02:13:24.880877 4328 raft_consensus.cc:348] T d2a8d96585fd4b4881d1bd028b1a9d36 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:24.881163 4328 raft_consensus.cc:374] T d2a8d96585fd4b4881d1bd028b1a9d36 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:24.881217 4328 raft_consensus.cc:729] T d2a8d96585fd4b4881d1bd028b1a9d36 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:13:24.881423 4328 consensus_queue.cc:260] T d2a8d96585fd4b4881d1bd028b1a9d36 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:24.881958 4328 ts_tablet_manager.cc:1398] T d2a8d96585fd4b4881d1bd028b1a9d36 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:24.884553 3795 raft_consensus.cc:482] T ebcd2c6e1722442fbe603cfcccb56ab0 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:24.884816 3795 raft_consensus.cc:504] T ebcd2c6e1722442fbe603cfcccb56ab0 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:24.885844 3795 leader_election.cc:288] T ebcd2c6e1722442fbe603cfcccb56ab0 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:24.893541 30133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcd2c6e1722442fbe603cfcccb56ab0" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:13:24.893880 30133 raft_consensus.cc:2454] T ebcd2c6e1722442fbe603cfcccb56ab0 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:13:24.897042 32708 leader_election.cc:302] T ebcd2c6e1722442fbe603cfcccb56ab0 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:13:24.897655 4331 tablet_bootstrap.cc:492] T d2a8d96585fd4b4881d1bd028b1a9d36 P 8a666e798fe8477d8dd1e51b385d5ead: No bootstrap required, opened a new log
I20240503 02:13:24.897848 4331 ts_tablet_manager.cc:1367] T d2a8d96585fd4b4881d1bd028b1a9d36 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.081s user 0.004s sys 0.007s
I20240503 02:13:24.897907 3821 raft_consensus.cc:2781] T ebcd2c6e1722442fbe603cfcccb56ab0 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:24.898082 3821 raft_consensus.cc:482] T ebcd2c6e1722442fbe603cfcccb56ab0 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:24.898202 3821 raft_consensus.cc:3037] T ebcd2c6e1722442fbe603cfcccb56ab0 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:24.898722 4331 raft_consensus.cc:348] T d2a8d96585fd4b4881d1bd028b1a9d36 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:24.898984 4331 raft_consensus.cc:374] T d2a8d96585fd4b4881d1bd028b1a9d36 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:24.899065 4331 raft_consensus.cc:729] T d2a8d96585fd4b4881d1bd028b1a9d36 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:24.899299 4331 consensus_queue.cc:260] T d2a8d96585fd4b4881d1bd028b1a9d36 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:24.902109 433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcd2c6e1722442fbe603cfcccb56ab0" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:13:24.902371 4329 tablet_bootstrap.cc:492] T d2a8d96585fd4b4881d1bd028b1a9d36 P 1f10f80f13c84e80b03eda7085a3b012: No bootstrap required, opened a new log
I20240503 02:13:24.902410 433 raft_consensus.cc:2454] T ebcd2c6e1722442fbe603cfcccb56ab0 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 0.
I20240503 02:13:24.902515 4329 ts_tablet_manager.cc:1367] T d2a8d96585fd4b4881d1bd028b1a9d36 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.053s user 0.011s sys 0.000s
I20240503 02:13:24.903194 4329 raft_consensus.cc:348] T d2a8d96585fd4b4881d1bd028b1a9d36 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:24.903388 4329 raft_consensus.cc:374] T d2a8d96585fd4b4881d1bd028b1a9d36 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:24.903435 4329 raft_consensus.cc:729] T d2a8d96585fd4b4881d1bd028b1a9d36 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:13:24.903616 4329 consensus_queue.cc:260] T d2a8d96585fd4b4881d1bd028b1a9d36 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:24.905915 4329 ts_tablet_manager.cc:1398] T d2a8d96585fd4b4881d1bd028b1a9d36 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:13:24.906100 4331 ts_tablet_manager.cc:1398] T d2a8d96585fd4b4881d1bd028b1a9d36 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:13:24.915375 3821 raft_consensus.cc:504] T ebcd2c6e1722442fbe603cfcccb56ab0 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:24.916024 3821 leader_election.cc:288] T ebcd2c6e1722442fbe603cfcccb56ab0 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Requested vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:24.916599 30133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcd2c6e1722442fbe603cfcccb56ab0" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:13:24.916813 30133 raft_consensus.cc:3037] T ebcd2c6e1722442fbe603cfcccb56ab0 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:24.920691 433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcd2c6e1722442fbe603cfcccb56ab0" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:13:24.920923 433 raft_consensus.cc:3037] T ebcd2c6e1722442fbe603cfcccb56ab0 P fbbc2be6382e47908ecb7c68c367bd26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:24.924043 433 raft_consensus.cc:2454] T ebcd2c6e1722442fbe603cfcccb56ab0 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:13:24.928906 30133 raft_consensus.cc:2454] T ebcd2c6e1722442fbe603cfcccb56ab0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:13:24.939193 32708 leader_election.cc:302] T ebcd2c6e1722442fbe603cfcccb56ab0 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:13:24.939697 3795 raft_consensus.cc:2781] T ebcd2c6e1722442fbe603cfcccb56ab0 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:24.939834 3795 raft_consensus.cc:686] T ebcd2c6e1722442fbe603cfcccb56ab0 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Becoming Leader. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Running, Role: LEADER
I20240503 02:13:24.940136 3795 consensus_queue.cc:237] T ebcd2c6e1722442fbe603cfcccb56ab0 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:24.956614 29729 catalog_manager.cc:5484] T ebcd2c6e1722442fbe603cfcccb56ab0 P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6). New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:24.971573 29872 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "58e1aaecf47944569296fc1e0780efb5"
dest_uuid: "16440e365430405fb10366ab70e2b429"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52924
I20240503 02:13:24.971861 29872 raft_consensus.cc:595] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:13:25.123437 4347 raft_consensus.cc:979] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429: : Instructing follower 9a4d92bc25374fb3960fde7d10be309d to start an election
I20240503 02:13:25.123610 4347 raft_consensus.cc:1067] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Signalling peer 9a4d92bc25374fb3960fde7d10be309d to start an election
I20240503 02:13:25.124594 32641 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "58e1aaecf47944569296fc1e0780efb5"
dest_uuid: "9a4d92bc25374fb3960fde7d10be309d"
from {username='slave'} at 127.22.108.1:43971
I20240503 02:13:25.124856 32641 raft_consensus.cc:482] T 58e1aaecf47944569296fc1e0780efb5 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:25.125005 32641 raft_consensus.cc:3037] T 58e1aaecf47944569296fc1e0780efb5 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:25.129302 32641 raft_consensus.cc:504] T 58e1aaecf47944569296fc1e0780efb5 P 9a4d92bc25374fb3960fde7d10be309d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:25.130658 433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58e1aaecf47944569296fc1e0780efb5" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:13:25.130695 29872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58e1aaecf47944569296fc1e0780efb5" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:13:25.130882 29872 raft_consensus.cc:3032] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:25.130905 433 raft_consensus.cc:3037] T 58e1aaecf47944569296fc1e0780efb5 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:25.130947 29872 raft_consensus.cc:729] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Running, Role: LEADER
I20240503 02:13:25.131253 29872 consensus_queue.cc:260] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:25.131717 32641 leader_election.cc:288] T 58e1aaecf47944569296fc1e0780efb5 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 2 election: Requested vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:13:25.131711 29872 raft_consensus.cc:3037] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:25.134567 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58e1aaecf47944569296fc1e0780efb5" candidate_uuid: "9a4d92bc25374fb3960fde7d10be309d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:13:25.134773 30264 raft_consensus.cc:3037] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:25.136638 433 raft_consensus.cc:2454] T 58e1aaecf47944569296fc1e0780efb5 P fbbc2be6382e47908ecb7c68c367bd26 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 2.
I20240503 02:13:25.137478 30264 raft_consensus.cc:2454] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 2.
I20240503 02:13:25.142206 32574 leader_election.cc:302] T 58e1aaecf47944569296fc1e0780efb5 P 9a4d92bc25374fb3960fde7d10be309d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, 9a4d92bc25374fb3960fde7d10be309d, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:13:25.142530 4147 raft_consensus.cc:2781] T 58e1aaecf47944569296fc1e0780efb5 P 9a4d92bc25374fb3960fde7d10be309d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:25.142712 4147 raft_consensus.cc:686] T 58e1aaecf47944569296fc1e0780efb5 P 9a4d92bc25374fb3960fde7d10be309d [term 2 LEADER]: Becoming Leader. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Running, Role: LEADER
I20240503 02:13:25.143013 4147 consensus_queue.cc:237] T 58e1aaecf47944569296fc1e0780efb5 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:25.146165 29872 raft_consensus.cc:2454] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4d92bc25374fb3960fde7d10be309d in term 2.
I20240503 02:13:25.146545 4347 raft_consensus.cc:482] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:25.146755 4347 raft_consensus.cc:504] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:25.147653 4347 leader_election.cc:288] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:13:25.148061 565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3533364f91c3433a8c1536b3eaee7424" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:13:25.148324 565 raft_consensus.cc:2454] T 3533364f91c3433a8c1536b3eaee7424 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:13:25.148684 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3533364f91c3433a8c1536b3eaee7424" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:13:25.148885 30264 raft_consensus.cc:2454] T 3533364f91c3433a8c1536b3eaee7424 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 0.
I20240503 02:13:25.151384 29808 leader_election.cc:302] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:25.152629 29731 catalog_manager.cc:5484] T 58e1aaecf47944569296fc1e0780efb5 P 9a4d92bc25374fb3960fde7d10be309d reported cstate change: term changed from 1 to 2, leader changed from 16440e365430405fb10366ab70e2b429 (127.22.108.1) to 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5). New cstate: current_term: 2 leader_uuid: "9a4d92bc25374fb3960fde7d10be309d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:25.154587 4347 raft_consensus.cc:2781] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:25.154682 4347 raft_consensus.cc:482] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:25.154739 4347 raft_consensus.cc:3037] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:25.158668 4347 raft_consensus.cc:504] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:25.164569 565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3533364f91c3433a8c1536b3eaee7424" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:13:25.164856 565 raft_consensus.cc:3037] T 3533364f91c3433a8c1536b3eaee7424 P f575ce92c0224eeb80b9fae5f6561fbe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:25.170998 565 raft_consensus.cc:2454] T 3533364f91c3433a8c1536b3eaee7424 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:13:25.171999 29808 leader_election.cc:302] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:13:25.172271 3498 raft_consensus.cc:2781] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:25.172408 3498 raft_consensus.cc:686] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Becoming Leader. State: Replica: 16440e365430405fb10366ab70e2b429, State: Running, Role: LEADER
I20240503 02:13:25.172624 4347 leader_election.cc:288] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 1 election: Requested vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:13:25.172660 3498 consensus_queue.cc:237] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:25.175288 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3533364f91c3433a8c1536b3eaee7424" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:13:25.175488 30264 raft_consensus.cc:3037] T 3533364f91c3433a8c1536b3eaee7424 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:25.181207 30264 raft_consensus.cc:2454] T 3533364f91c3433a8c1536b3eaee7424 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:13:25.212488 29730 catalog_manager.cc:5484] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16440e365430405fb10366ab70e2b429 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:25.230527 3899 raft_consensus.cc:482] T d2a8d96585fd4b4881d1bd028b1a9d36 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:25.230697 3899 raft_consensus.cc:504] T d2a8d96585fd4b4881d1bd028b1a9d36 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:25.231356 3899 leader_election.cc:288] T d2a8d96585fd4b4881d1bd028b1a9d36 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:25.232131 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a8d96585fd4b4881d1bd028b1a9d36" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:13:25.232337 30266 raft_consensus.cc:2454] T d2a8d96585fd4b4881d1bd028b1a9d36 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 0.
I20240503 02:13:25.233670 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a8d96585fd4b4881d1bd028b1a9d36" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:13:25.233884 307 raft_consensus.cc:2454] T d2a8d96585fd4b4881d1bd028b1a9d36 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 0.
I20240503 02:13:25.234704 30070 leader_election.cc:302] T d2a8d96585fd4b4881d1bd028b1a9d36 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:13:25.234997 3899 raft_consensus.cc:2781] T d2a8d96585fd4b4881d1bd028b1a9d36 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:25.235080 3899 raft_consensus.cc:482] T d2a8d96585fd4b4881d1bd028b1a9d36 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:25.235136 3899 raft_consensus.cc:3037] T d2a8d96585fd4b4881d1bd028b1a9d36 P a9094240860e43bfb129193f3bfd5be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:25.237825 3899 raft_consensus.cc:504] T d2a8d96585fd4b4881d1bd028b1a9d36 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:25.238646 3899 leader_election.cc:288] T d2a8d96585fd4b4881d1bd028b1a9d36 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Requested vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:25.241271 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a8d96585fd4b4881d1bd028b1a9d36" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:13:25.241482 30266 raft_consensus.cc:3037] T d2a8d96585fd4b4881d1bd028b1a9d36 P 1f10f80f13c84e80b03eda7085a3b012 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:25.244212 30266 raft_consensus.cc:2454] T d2a8d96585fd4b4881d1bd028b1a9d36 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 1.
I20240503 02:13:25.244719 30070 leader_election.cc:302] T d2a8d96585fd4b4881d1bd028b1a9d36 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:13:25.245190 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a8d96585fd4b4881d1bd028b1a9d36" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:13:25.245409 307 raft_consensus.cc:3037] T d2a8d96585fd4b4881d1bd028b1a9d36 P 8a666e798fe8477d8dd1e51b385d5ead [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:25.247853 307 raft_consensus.cc:2454] T d2a8d96585fd4b4881d1bd028b1a9d36 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 1.
I20240503 02:13:25.248255 3899 raft_consensus.cc:2781] T d2a8d96585fd4b4881d1bd028b1a9d36 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:25.248389 3899 raft_consensus.cc:686] T d2a8d96585fd4b4881d1bd028b1a9d36 P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Becoming Leader. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Running, Role: LEADER
I20240503 02:13:25.249791 4339 consensus_queue.cc:1035] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:25.262429 3899 consensus_queue.cc:237] T d2a8d96585fd4b4881d1bd028b1a9d36 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:25.267169 4091 consensus_queue.cc:1035] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:25.273149 29727 catalog_manager.cc:5484] T d2a8d96585fd4b4881d1bd028b1a9d36 P a9094240860e43bfb129193f3bfd5be6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9094240860e43bfb129193f3bfd5be6 (127.22.108.3). New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:25.387532 4355 consensus_queue.cc:1035] T ebcd2c6e1722442fbe603cfcccb56ab0 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:25.452572 29853 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:25.460394 30244 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:25.493144 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:25.499044 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:25.506736 4355 consensus_queue.cc:1035] T ebcd2c6e1722442fbe603cfcccb56ab0 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:25.513347 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:25.526641 29874 raft_consensus.cc:1261] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429 [term 2 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:13:25.527252 4147 consensus_queue.cc:1035] T 58e1aaecf47944569296fc1e0780efb5 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:25.561945 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:25.573704 30266 raft_consensus.cc:1261] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:13:25.574332 4147 consensus_queue.cc:1035] T 58e1aaecf47944569296fc1e0780efb5 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:25.614553 434 raft_consensus.cc:1261] T 58e1aaecf47944569296fc1e0780efb5 P fbbc2be6382e47908ecb7c68c367bd26 [term 2 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:13:25.615294 4147 consensus_queue.cc:1035] T 58e1aaecf47944569296fc1e0780efb5 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:25.637392 32755 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:25.676910 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:25.702024 3498 consensus_queue.cc:1035] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:25.709348 3498 consensus_queue.cc:1035] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:25.754264 4322 consensus_queue.cc:1035] T d2a8d96585fd4b4881d1bd028b1a9d36 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:25.759243 3899 consensus_queue.cc:1035] T d2a8d96585fd4b4881d1bd028b1a9d36 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:26.243381 29853 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:26.265952 30244 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:26.266659 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 02:13:26.309121 32708 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.8:42233, user_credentials={real_user=slave}} blocked reactor thread for 56233.5us
I20240503 02:13:26.354460 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:26.382676 32755 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:26.430487 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 02:13:26.454452 30070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.5:35595, user_credentials={real_user=slave}} blocked reactor thread for 52490.8us
I20240503 02:13:26.543131 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:26.578101 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:26.691852 30133 consensus_queue.cc:237] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:26.694247 30266 raft_consensus.cc:1261] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:26.694269 32641 raft_consensus.cc:1261] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:26.694794 3899 consensus_queue.cc:1035] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:26.695271 4551 consensus_queue.cc:1035] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:26.698525 3899 raft_consensus.cc:2932] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:26.699240 30266 raft_consensus.cc:2932] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:26.706769 29730 catalog_manager.cc:5484] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: config changed from index 4 to 5, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) evicted. New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:26.709949 32641 raft_consensus.cc:2932] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:26.721743 29985 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fdbaf1df10b46d5a47326415d0cd0e9 with delete_type TABLET_DATA_TOMBSTONED (TS 155735c7f8b14997ac4c770aee2a9401 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:52158
I20240503 02:13:26.729642 4582 tablet_replica.cc:330] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:13:26.729938 4582 raft_consensus.cc:2229] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:26.730201 4582 raft_consensus.cc:2258] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:26.730906 4582 ts_tablet_manager.cc:1875] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:26.737048 4582 ts_tablet_manager.cc:1888] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:13:26.737655 4582 log.cc:1196] T 8fdbaf1df10b46d5a47326415d0cd0e9 P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/8fdbaf1df10b46d5a47326415d0cd0e9
I20240503 02:13:26.738560 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet 8fdbaf1df10b46d5a47326415d0cd0e9 (table rebalance_test_table_5 [id=99dbec2a2b63467ab34b96ecc8479244]) successfully deleted
I20240503 02:13:27.133203 29853 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:27.134410 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:27.172441 30244 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:27.185043 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:27.187583 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:27.219928 32755 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:27.243137 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:27.262969 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:27.423403 565 consensus_queue.cc:237] T 7527cc5931e24fa180efedbf81438efd P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:27.425746 29874 raft_consensus.cc:1261] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 [term 2 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:13:27.425989 30132 raft_consensus.cc:1261] T 7527cc5931e24fa180efedbf81438efd P a9094240860e43bfb129193f3bfd5be6 [term 2 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:13:27.426366 4585 consensus_queue.cc:1035] T 7527cc5931e24fa180efedbf81438efd P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:27.426689 4409 consensus_queue.cc:1035] T 7527cc5931e24fa180efedbf81438efd P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:27.434469 4409 raft_consensus.cc:2932] T 7527cc5931e24fa180efedbf81438efd P f575ce92c0224eeb80b9fae5f6561fbe [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:27.437964 29874 raft_consensus.cc:2932] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:27.444728 29726 catalog_manager.cc:5484] T 7527cc5931e24fa180efedbf81438efd P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: config changed from index 4 to 6, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) evicted. New cstate: current_term: 2 leader_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:27.455459 30244 tablet_service.cc:1514] Processing DeleteTablet for tablet 7527cc5931e24fa180efedbf81438efd with delete_type TABLET_DATA_TOMBSTONED (TS 1f10f80f13c84e80b03eda7085a3b012 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:49856
I20240503 02:13:27.460062 30132 raft_consensus.cc:2932] T 7527cc5931e24fa180efedbf81438efd P a9094240860e43bfb129193f3bfd5be6 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:27.478169 4674 tablet_replica.cc:330] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012: stopping tablet replica
I20240503 02:13:27.478443 4674 raft_consensus.cc:2229] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:27.478646 4674 raft_consensus.cc:2258] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:27.479423 4674 ts_tablet_manager.cc:1875] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:27.488775 4674 ts_tablet_manager.cc:1888] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 02:13:27.488926 4674 log.cc:1196] T 7527cc5931e24fa180efedbf81438efd P 1f10f80f13c84e80b03eda7085a3b012: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/wals/7527cc5931e24fa180efedbf81438efd
I20240503 02:13:27.489722 29719 catalog_manager.cc:4830] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): tablet 7527cc5931e24fa180efedbf81438efd (table rebalance_test_table_3 [id=3131069dba4a4f9aa26d4c134fcb4704]) successfully deleted
I20240503 02:13:27.783298 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:27.788130 32755 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:27.792836 29853 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:27.829668 30244 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:27.839638 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:27.850112 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:27.898451 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:27.967638 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:28.120793 29874 consensus_queue.cc:237] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:28.122931 565 raft_consensus.cc:1261] T 493c813acc27492fb952791cf2dc12a2 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:28.123109 32641 raft_consensus.cc:1261] T 493c813acc27492fb952791cf2dc12a2 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:28.123442 4673 consensus_queue.cc:1035] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:28.124100 4673 consensus_queue.cc:1035] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:28.127907 4444 raft_consensus.cc:2932] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:28.128455 565 raft_consensus.cc:2932] T 493c813acc27492fb952791cf2dc12a2 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:28.129123 32641 raft_consensus.cc:2932] T 493c813acc27492fb952791cf2dc12a2 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:28.135290 29726 catalog_manager.cc:5484] T 493c813acc27492fb952791cf2dc12a2 P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: config changed from index 2 to 3, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) evicted. New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:28.148418 29985 tablet_service.cc:1514] Processing DeleteTablet for tablet 493c813acc27492fb952791cf2dc12a2 with delete_type TABLET_DATA_TOMBSTONED (TS 155735c7f8b14997ac4c770aee2a9401 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:52158
I20240503 02:13:28.160805 4763 tablet_replica.cc:330] T 493c813acc27492fb952791cf2dc12a2 P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:13:28.161055 4763 raft_consensus.cc:2229] T 493c813acc27492fb952791cf2dc12a2 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:28.161280 4763 raft_consensus.cc:2258] T 493c813acc27492fb952791cf2dc12a2 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:28.161909 4763 ts_tablet_manager.cc:1875] T 493c813acc27492fb952791cf2dc12a2 P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:28.168264 4763 ts_tablet_manager.cc:1888] T 493c813acc27492fb952791cf2dc12a2 P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:13:28.168370 4763 log.cc:1196] T 493c813acc27492fb952791cf2dc12a2 P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/493c813acc27492fb952791cf2dc12a2
I20240503 02:13:28.169207 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet 493c813acc27492fb952791cf2dc12a2 (table rebalancer_extra_table_40 [id=2285665685314d5cae1699d9f2c791f0]) successfully deleted
I20240503 02:13:28.481529 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:28.506402 29853 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:28.533314 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:28.565976 30244 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:28.592448 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:28.602310 32755 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:28.613150 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:28.619390 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:28.779976 30135 consensus_queue.cc:237] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:28.782164 565 raft_consensus.cc:1261] T e4fe8ae88ae843609a66c7a2e6c7b4af P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:28.782222 30005 raft_consensus.cc:1261] T e4fe8ae88ae843609a66c7a2e6c7b4af P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:28.783058 3899 consensus_queue.cc:1035] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:28.783361 4551 consensus_queue.cc:1035] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:28.786168 3899 raft_consensus.cc:2932] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:28.786762 565 raft_consensus.cc:2932] T e4fe8ae88ae843609a66c7a2e6c7b4af P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:28.786772 30005 raft_consensus.cc:2932] T e4fe8ae88ae843609a66c7a2e6c7b4af P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:28.792840 29725 catalog_manager.cc:5484] T e4fe8ae88ae843609a66c7a2e6c7b4af P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: config changed from index 2 to 3, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:28.798739 32755 tablet_service.cc:1514] Processing DeleteTablet for tablet e4fe8ae88ae843609a66c7a2e6c7b4af with delete_type TABLET_DATA_TOMBSTONED (TS 8a666e798fe8477d8dd1e51b385d5ead not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46360
I20240503 02:13:28.799716 4858 tablet_replica.cc:330] T e4fe8ae88ae843609a66c7a2e6c7b4af P 8a666e798fe8477d8dd1e51b385d5ead: stopping tablet replica
I20240503 02:13:28.800009 4858 raft_consensus.cc:2229] T e4fe8ae88ae843609a66c7a2e6c7b4af P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:28.800416 4858 raft_consensus.cc:2258] T e4fe8ae88ae843609a66c7a2e6c7b4af P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:28.801086 4858 ts_tablet_manager.cc:1875] T e4fe8ae88ae843609a66c7a2e6c7b4af P 8a666e798fe8477d8dd1e51b385d5ead: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:28.807869 4858 ts_tablet_manager.cc:1888] T e4fe8ae88ae843609a66c7a2e6c7b4af P 8a666e798fe8477d8dd1e51b385d5ead: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:13:28.808004 4858 log.cc:1196] T e4fe8ae88ae843609a66c7a2e6c7b4af P 8a666e798fe8477d8dd1e51b385d5ead: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal/wals/e4fe8ae88ae843609a66c7a2e6c7b4af
I20240503 02:13:28.808737 29719 catalog_manager.cc:4830] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): tablet e4fe8ae88ae843609a66c7a2e6c7b4af (table rebalancer_extra_table_14 [id=098275b181bb4441990148dfd09ded67]) successfully deleted
I20240503 02:13:28.915405 30005 consensus_queue.cc:237] T 63b416f4e004490b8e38a96fd065dfea P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:28.917737 30135 raft_consensus.cc:1261] T 63b416f4e004490b8e38a96fd065dfea P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:28.917804 29874 raft_consensus.cc:1261] T 63b416f4e004490b8e38a96fd065dfea P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:13:28.918138 29940 consensus_peers.cc:485] T 63b416f4e004490b8e38a96fd065dfea P 155735c7f8b14997ac4c770aee2a9401 -> Peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): Couldn't send request to peer fbbc2be6382e47908ecb7c68c367bd26. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63b416f4e004490b8e38a96fd065dfea. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:28.918306 4646 consensus_queue.cc:1035] T 63b416f4e004490b8e38a96fd065dfea P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:28.918771 4646 consensus_queue.cc:1035] T 63b416f4e004490b8e38a96fd065dfea P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:28.934309 4437 raft_consensus.cc:2932] T 63b416f4e004490b8e38a96fd065dfea P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:28.938905 29874 raft_consensus.cc:2932] T 63b416f4e004490b8e38a96fd065dfea P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:28.941254 30135 raft_consensus.cc:2932] T 63b416f4e004490b8e38a96fd065dfea P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:28.941625 29729 catalog_manager.cc:5484] T 63b416f4e004490b8e38a96fd065dfea P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: config changed from index -1 to 4, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:28.999704 32638 consensus_queue.cc:237] T 0ec95509de3443bd96bf91a08ef7df4a P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:29.003029 565 raft_consensus.cc:1261] T 0ec95509de3443bd96bf91a08ef7df4a P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:29.003562 4147 consensus_queue.cc:1035] T 0ec95509de3443bd96bf91a08ef7df4a P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:13:29.004027 32576 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 9a4d92bc25374fb3960fde7d10be309d -> Peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Couldn't send request to peer 16440e365430405fb10366ab70e2b429. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0ec95509de3443bd96bf91a08ef7df4a. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:29.006196 305 raft_consensus.cc:1261] T 0ec95509de3443bd96bf91a08ef7df4a P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:29.006829 4147 consensus_queue.cc:1035] T 0ec95509de3443bd96bf91a08ef7df4a P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:29.009622 4812 raft_consensus.cc:2932] T 0ec95509de3443bd96bf91a08ef7df4a P 9a4d92bc25374fb3960fde7d10be309d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:29.013682 305 raft_consensus.cc:2932] T 0ec95509de3443bd96bf91a08ef7df4a P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:29.017985 565 raft_consensus.cc:2932] T 0ec95509de3443bd96bf91a08ef7df4a P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:29.025780 29729 catalog_manager.cc:5484] T 0ec95509de3443bd96bf91a08ef7df4a P 9a4d92bc25374fb3960fde7d10be309d reported cstate change: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New cstate: current_term: 1 leader_uuid: "9a4d92bc25374fb3960fde7d10be309d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:29.058233 29874 consensus_queue.cc:237] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
W20240503 02:13:29.062583 29808 consensus_peers.cc:485] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 16440e365430405fb10366ab70e2b429 -> Peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Couldn't send request to peer a9094240860e43bfb129193f3bfd5be6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 70b8ebcf6f3245e6bd93eeb38ca1cbf5. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:29.063021 565 raft_consensus.cc:1261] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:29.063306 30005 raft_consensus.cc:1261] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:29.063558 4444 consensus_queue.cc:1035] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:29.064309 4444 consensus_queue.cc:1035] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:29.067013 4927 raft_consensus.cc:2932] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:29.067514 30005 raft_consensus.cc:2932] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:29.071127 565 raft_consensus.cc:2932] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:29.073668 29726 catalog_manager.cc:5484] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:29.112025 29874 consensus_queue.cc:237] T 1758ec3783b24d3c820ed0d6dc118e69 P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:29.114383 30004 raft_consensus.cc:1261] T 1758ec3783b24d3c820ed0d6dc118e69 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:29.114923 4927 consensus_queue.cc:1035] T 1758ec3783b24d3c820ed0d6dc118e69 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:13:29.115219 29809 consensus_peers.cc:485] T 1758ec3783b24d3c820ed0d6dc118e69 P 16440e365430405fb10366ab70e2b429 -> Peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): Couldn't send request to peer 1f10f80f13c84e80b03eda7085a3b012. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1758ec3783b24d3c820ed0d6dc118e69. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:29.115244 434 raft_consensus.cc:1261] T 1758ec3783b24d3c820ed0d6dc118e69 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:29.115725 4927 consensus_queue.cc:1035] T 1758ec3783b24d3c820ed0d6dc118e69 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:29.120525 4444 raft_consensus.cc:2932] T 1758ec3783b24d3c820ed0d6dc118e69 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:29.121317 434 raft_consensus.cc:2932] T 1758ec3783b24d3c820ed0d6dc118e69 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:29.124923 30004 raft_consensus.cc:2932] T 1758ec3783b24d3c820ed0d6dc118e69 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:29.127166 29725 catalog_manager.cc:5484] T 1758ec3783b24d3c820ed0d6dc118e69 P 16440e365430405fb10366ab70e2b429 reported cstate change: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:29.187950 434 consensus_queue.cc:237] T ec3891bfc4014b63afef6c259de00ca7 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:29.190325 306 raft_consensus.cc:1261] T ec3891bfc4014b63afef6c259de00ca7 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:29.190843 32638 raft_consensus.cc:1261] T ec3891bfc4014b63afef6c259de00ca7 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:29.190948 4965 consensus_queue.cc:1035] T ec3891bfc4014b63afef6c259de00ca7 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:29.191620 4830 consensus_queue.cc:1035] T ec3891bfc4014b63afef6c259de00ca7 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:13:29.193502 373 consensus_peers.cc:485] T ec3891bfc4014b63afef6c259de00ca7 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Couldn't send request to peer 16440e365430405fb10366ab70e2b429. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ec3891bfc4014b63afef6c259de00ca7. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:29.195526 4965 raft_consensus.cc:2932] T ec3891bfc4014b63afef6c259de00ca7 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:29.196192 305 raft_consensus.cc:2932] T ec3891bfc4014b63afef6c259de00ca7 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:29.199797 32638 raft_consensus.cc:2932] T ec3891bfc4014b63afef6c259de00ca7 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:29.203256 29726 catalog_manager.cc:5484] T ec3891bfc4014b63afef6c259de00ca7 P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:29.244383 305 consensus_queue.cc:237] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:29.246968 30266 raft_consensus.cc:1261] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:29.247166 29874 raft_consensus.cc:1261] T f694fefb42d84318a6d91532105b3c90 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:13:29.247493 32708 consensus_peers.cc:485] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f694fefb42d84318a6d91532105b3c90. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:29.247759 4547 consensus_queue.cc:1035] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:29.248108 4985 consensus_queue.cc:1035] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:29.254256 4985 raft_consensus.cc:2932] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:29.254874 29874 raft_consensus.cc:2932] T f694fefb42d84318a6d91532105b3c90 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:29.258697 30266 raft_consensus.cc:2932] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:29.260768 29725 catalog_manager.cc:5484] T f694fefb42d84318a6d91532105b3c90 P 16440e365430405fb10366ab70e2b429 reported cstate change: config changed from index -1 to 2, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:29.296877 30004 consensus_queue.cc:237] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:29.299777 30135 raft_consensus.cc:1261] T 04e47da8cf7445dcacaf1f7c8950bd34 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:29.299932 30266 raft_consensus.cc:1261] T 04e47da8cf7445dcacaf1f7c8950bd34 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:13:29.300154 29940 consensus_peers.cc:485] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 -> Peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): Couldn't send request to peer fbbc2be6382e47908ecb7c68c367bd26. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04e47da8cf7445dcacaf1f7c8950bd34. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:29.300485 4437 consensus_queue.cc:1035] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:29.300755 4646 consensus_queue.cc:1035] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:29.303344 4646 raft_consensus.cc:2932] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:29.306519 30135 raft_consensus.cc:2932] T 04e47da8cf7445dcacaf1f7c8950bd34 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:29.307193 30266 raft_consensus.cc:2932] T 04e47da8cf7445dcacaf1f7c8950bd34 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:29.309293 29729 catalog_manager.cc:5484] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: config changed from index -1 to 4, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:29.347357 305 consensus_queue.cc:237] T 00aedc4902a745d6a859cbbf99529a7f P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:29.349900 30266 raft_consensus.cc:1261] T 00aedc4902a745d6a859cbbf99529a7f P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:29.350438 4547 consensus_queue.cc:1035] T 00aedc4902a745d6a859cbbf99529a7f P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:29.350724 32638 raft_consensus.cc:1261] T 00aedc4902a745d6a859cbbf99529a7f P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:29.351279 4547 consensus_queue.cc:1035] T 00aedc4902a745d6a859cbbf99529a7f P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:13:29.356313 32709 consensus_peers.cc:485] T 00aedc4902a745d6a859cbbf99529a7f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Couldn't send request to peer 155735c7f8b14997ac4c770aee2a9401. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00aedc4902a745d6a859cbbf99529a7f. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:29.366802 5024 ts_tablet_manager.cc:902] T 63b416f4e004490b8e38a96fd065dfea P fbbc2be6382e47908ecb7c68c367bd26: Initiating tablet copy from peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:13:29.380646 4985 raft_consensus.cc:2932] T 00aedc4902a745d6a859cbbf99529a7f P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:29.381348 32641 raft_consensus.cc:2932] T 00aedc4902a745d6a859cbbf99529a7f P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:29.388940 29725 catalog_manager.cc:5484] T 00aedc4902a745d6a859cbbf99529a7f P 9a4d92bc25374fb3960fde7d10be309d reported cstate change: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:29.389222 30266 raft_consensus.cc:2932] T 00aedc4902a745d6a859cbbf99529a7f P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:29.421012 5024 tablet_copy_client.cc:308] T 63b416f4e004490b8e38a96fd065dfea P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.2:37873
I20240503 02:13:29.421976 30026 tablet_copy_service.cc:140] P 155735c7f8b14997ac4c770aee2a9401: Received BeginTabletCopySession request for tablet 63b416f4e004490b8e38a96fd065dfea from peer fbbc2be6382e47908ecb7c68c367bd26 ({username='slave'} at 127.22.108.7:60369)
I20240503 02:13:29.422137 30026 tablet_copy_service.cc:161] P 155735c7f8b14997ac4c770aee2a9401: Beginning new tablet copy session on tablet 63b416f4e004490b8e38a96fd065dfea from peer fbbc2be6382e47908ecb7c68c367bd26 at {username='slave'} at 127.22.108.7:60369: session id = fbbc2be6382e47908ecb7c68c367bd26-63b416f4e004490b8e38a96fd065dfea
I20240503 02:13:29.424949 30026 tablet_copy_source_session.cc:205] T 63b416f4e004490b8e38a96fd065dfea P 155735c7f8b14997ac4c770aee2a9401: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:29.428448 5024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63b416f4e004490b8e38a96fd065dfea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:29.441406 5024 tablet_copy_client.cc:783] T 63b416f4e004490b8e38a96fd065dfea P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:29.441665 5024 tablet_copy_client.cc:647] T 63b416f4e004490b8e38a96fd065dfea P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:29.444928 5024 tablet_copy_client.cc:523] T 63b416f4e004490b8e38a96fd065dfea P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:29.449992 5024 tablet_bootstrap.cc:492] T 63b416f4e004490b8e38a96fd065dfea P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:29.524873 5037 ts_tablet_manager.cc:902] T 1758ec3783b24d3c820ed0d6dc118e69 P 1f10f80f13c84e80b03eda7085a3b012: Initiating tablet copy from peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:13:29.531760 5024 tablet_bootstrap.cc:492] T 63b416f4e004490b8e38a96fd065dfea P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:29.532253 5024 tablet_bootstrap.cc:492] T 63b416f4e004490b8e38a96fd065dfea P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap complete.
I20240503 02:13:29.536391 5024 ts_tablet_manager.cc:1367] T 63b416f4e004490b8e38a96fd065dfea P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.086s user 0.053s sys 0.010s
I20240503 02:13:29.537256 5024 raft_consensus.cc:348] T 63b416f4e004490b8e38a96fd065dfea P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:29.537496 5024 raft_consensus.cc:729] T 63b416f4e004490b8e38a96fd065dfea P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:29.539242 5024 consensus_queue.cc:260] T 63b416f4e004490b8e38a96fd065dfea P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:29.540433 5024 ts_tablet_manager.cc:1398] T 63b416f4e004490b8e38a96fd065dfea P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:13:29.542208 30026 tablet_copy_service.cc:342] P 155735c7f8b14997ac4c770aee2a9401: Request end of tablet copy session fbbc2be6382e47908ecb7c68c367bd26-63b416f4e004490b8e38a96fd065dfea received from {username='slave'} at 127.22.108.7:60369
I20240503 02:13:29.542318 30026 tablet_copy_service.cc:433] P 155735c7f8b14997ac4c770aee2a9401: ending tablet copy session fbbc2be6382e47908ecb7c68c367bd26-63b416f4e004490b8e38a96fd065dfea on tablet 63b416f4e004490b8e38a96fd065dfea with peer fbbc2be6382e47908ecb7c68c367bd26
I20240503 02:13:29.550097 5041 ts_tablet_manager.cc:902] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P a9094240860e43bfb129193f3bfd5be6: Initiating tablet copy from peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:13:29.563045 5041 tablet_copy_client.cc:308] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P a9094240860e43bfb129193f3bfd5be6: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.1:36719
I20240503 02:13:29.563942 29895 tablet_copy_service.cc:140] P 16440e365430405fb10366ab70e2b429: Received BeginTabletCopySession request for tablet 70b8ebcf6f3245e6bd93eeb38ca1cbf5 from peer a9094240860e43bfb129193f3bfd5be6 ({username='slave'} at 127.22.108.3:49185)
I20240503 02:13:29.564085 29895 tablet_copy_service.cc:161] P 16440e365430405fb10366ab70e2b429: Beginning new tablet copy session on tablet 70b8ebcf6f3245e6bd93eeb38ca1cbf5 from peer a9094240860e43bfb129193f3bfd5be6 at {username='slave'} at 127.22.108.3:49185: session id = a9094240860e43bfb129193f3bfd5be6-70b8ebcf6f3245e6bd93eeb38ca1cbf5
I20240503 02:13:29.566941 29895 tablet_copy_source_session.cc:205] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 16440e365430405fb10366ab70e2b429: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:29.574939 5041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70b8ebcf6f3245e6bd93eeb38ca1cbf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:29.584848 5037 tablet_copy_client.cc:308] T 1758ec3783b24d3c820ed0d6dc118e69 P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.1:36719
I20240503 02:13:29.586119 29895 tablet_copy_service.cc:140] P 16440e365430405fb10366ab70e2b429: Received BeginTabletCopySession request for tablet 1758ec3783b24d3c820ed0d6dc118e69 from peer 1f10f80f13c84e80b03eda7085a3b012 ({username='slave'} at 127.22.108.4:51273)
I20240503 02:13:29.586258 29895 tablet_copy_service.cc:161] P 16440e365430405fb10366ab70e2b429: Beginning new tablet copy session on tablet 1758ec3783b24d3c820ed0d6dc118e69 from peer 1f10f80f13c84e80b03eda7085a3b012 at {username='slave'} at 127.22.108.4:51273: session id = 1f10f80f13c84e80b03eda7085a3b012-1758ec3783b24d3c820ed0d6dc118e69
I20240503 02:13:29.586849 5041 tablet_copy_client.cc:783] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P a9094240860e43bfb129193f3bfd5be6: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:29.587131 5041 tablet_copy_client.cc:647] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P a9094240860e43bfb129193f3bfd5be6: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:29.588201 29895 tablet_copy_source_session.cc:205] T 1758ec3783b24d3c820ed0d6dc118e69 P 16440e365430405fb10366ab70e2b429: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:29.588900 5041 tablet_copy_client.cc:523] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P a9094240860e43bfb129193f3bfd5be6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:29.589373 5037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1758ec3783b24d3c820ed0d6dc118e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:29.593271 5041 tablet_bootstrap.cc:492] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:13:29.596407 5037 tablet_copy_client.cc:783] T 1758ec3783b24d3c820ed0d6dc118e69 P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:29.596685 5037 tablet_copy_client.cc:647] T 1758ec3783b24d3c820ed0d6dc118e69 P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:29.598441 5037 tablet_copy_client.cc:523] T 1758ec3783b24d3c820ed0d6dc118e69 P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:29.602684 5052 ts_tablet_manager.cc:902] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429: Initiating tablet copy from peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:13:29.604218 5037 tablet_bootstrap.cc:492] T 1758ec3783b24d3c820ed0d6dc118e69 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:13:29.606704 5052 tablet_copy_client.cc:308] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.5:35595
I20240503 02:13:29.607796 32662 tablet_copy_service.cc:140] P 9a4d92bc25374fb3960fde7d10be309d: Received BeginTabletCopySession request for tablet 0ec95509de3443bd96bf91a08ef7df4a from peer 16440e365430405fb10366ab70e2b429 ({username='slave'} at 127.22.108.1:43971)
I20240503 02:13:29.608057 32662 tablet_copy_service.cc:161] P 9a4d92bc25374fb3960fde7d10be309d: Beginning new tablet copy session on tablet 0ec95509de3443bd96bf91a08ef7df4a from peer 16440e365430405fb10366ab70e2b429 at {username='slave'} at 127.22.108.1:43971: session id = 16440e365430405fb10366ab70e2b429-0ec95509de3443bd96bf91a08ef7df4a
I20240503 02:13:29.612622 32662 tablet_copy_source_session.cc:205] T 0ec95509de3443bd96bf91a08ef7df4a P 9a4d92bc25374fb3960fde7d10be309d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:29.614881 5052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ec95509de3443bd96bf91a08ef7df4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:29.622505 5041 tablet_bootstrap.cc:492] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P a9094240860e43bfb129193f3bfd5be6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:29.623302 5041 tablet_bootstrap.cc:492] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P a9094240860e43bfb129193f3bfd5be6: Bootstrap complete.
I20240503 02:13:29.623555 5041 ts_tablet_manager.cc:1367] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.030s user 0.018s sys 0.008s
I20240503 02:13:29.624133 5041 raft_consensus.cc:348] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:29.624291 5041 raft_consensus.cc:729] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:13:29.624508 5041 consensus_queue.cc:260] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:29.625061 5041 ts_tablet_manager.cc:1398] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:13:29.627460 5037 tablet_bootstrap.cc:492] T 1758ec3783b24d3c820ed0d6dc118e69 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:29.627779 5037 tablet_bootstrap.cc:492] T 1758ec3783b24d3c820ed0d6dc118e69 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap complete.
I20240503 02:13:29.627960 5037 ts_tablet_manager.cc:1367] T 1758ec3783b24d3c820ed0d6dc118e69 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.000s
I20240503 02:13:29.628512 5037 raft_consensus.cc:348] T 1758ec3783b24d3c820ed0d6dc118e69 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:29.628718 5037 raft_consensus.cc:729] T 1758ec3783b24d3c820ed0d6dc118e69 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:13:29.628968 5037 consensus_queue.cc:260] T 1758ec3783b24d3c820ed0d6dc118e69 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:29.629215 29895 tablet_copy_service.cc:342] P 16440e365430405fb10366ab70e2b429: Request end of tablet copy session a9094240860e43bfb129193f3bfd5be6-70b8ebcf6f3245e6bd93eeb38ca1cbf5 received from {username='slave'} at 127.22.108.3:49185
I20240503 02:13:29.629324 29895 tablet_copy_service.cc:433] P 16440e365430405fb10366ab70e2b429: ending tablet copy session a9094240860e43bfb129193f3bfd5be6-70b8ebcf6f3245e6bd93eeb38ca1cbf5 on tablet 70b8ebcf6f3245e6bd93eeb38ca1cbf5 with peer a9094240860e43bfb129193f3bfd5be6
I20240503 02:13:29.630402 5037 ts_tablet_manager.cc:1398] T 1758ec3783b24d3c820ed0d6dc118e69 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:13:29.631037 29895 tablet_copy_service.cc:342] P 16440e365430405fb10366ab70e2b429: Request end of tablet copy session 1f10f80f13c84e80b03eda7085a3b012-1758ec3783b24d3c820ed0d6dc118e69 received from {username='slave'} at 127.22.108.4:51273
I20240503 02:13:29.631142 29895 tablet_copy_service.cc:433] P 16440e365430405fb10366ab70e2b429: ending tablet copy session 1f10f80f13c84e80b03eda7085a3b012-1758ec3783b24d3c820ed0d6dc118e69 on tablet 1758ec3783b24d3c820ed0d6dc118e69 with peer 1f10f80f13c84e80b03eda7085a3b012
I20240503 02:13:29.637674 5052 tablet_copy_client.cc:783] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:29.638020 5052 tablet_copy_client.cc:647] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:29.640457 5052 tablet_copy_client.cc:523] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:29.643626 5052 tablet_bootstrap.cc:492] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:13:29.650519 5052 tablet_bootstrap.cc:492] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:29.650918 5052 tablet_bootstrap.cc:492] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429: Bootstrap complete.
I20240503 02:13:29.651235 5052 ts_tablet_manager.cc:1367] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20240503 02:13:29.651890 5052 raft_consensus.cc:348] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:29.652132 5052 raft_consensus.cc:729] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:13:29.652384 5052 consensus_queue.cc:260] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:29.653331 5052 ts_tablet_manager.cc:1398] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:13:29.655195 32662 tablet_copy_service.cc:342] P 9a4d92bc25374fb3960fde7d10be309d: Request end of tablet copy session 16440e365430405fb10366ab70e2b429-0ec95509de3443bd96bf91a08ef7df4a received from {username='slave'} at 127.22.108.1:43971
I20240503 02:13:29.655388 32662 tablet_copy_service.cc:433] P 9a4d92bc25374fb3960fde7d10be309d: ending tablet copy session 16440e365430405fb10366ab70e2b429-0ec95509de3443bd96bf91a08ef7df4a on tablet 0ec95509de3443bd96bf91a08ef7df4a with peer 16440e365430405fb10366ab70e2b429
I20240503 02:13:29.670668 30135 consensus_queue.cc:237] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:29.673187 30266 raft_consensus.cc:1261] T 5040203bd48549ff9c8f64c71d6bc598 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:29.673838 29874 raft_consensus.cc:1261] T 5040203bd48549ff9c8f64c71d6bc598 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:13:29.674225 30069 consensus_peers.cc:485] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5040203bd48549ff9c8f64c71d6bc598. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:29.674430 5068 consensus_queue.cc:1035] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:29.675550 5068 consensus_queue.cc:1035] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:29.679117 4551 raft_consensus.cc:2932] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:29.679723 30264 raft_consensus.cc:2932] T 5040203bd48549ff9c8f64c71d6bc598 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:29.686169 29725 catalog_manager.cc:5484] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 reported cstate change: config changed from index -1 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:29.688442 29874 raft_consensus.cc:2932] T 5040203bd48549ff9c8f64c71d6bc598 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:29.698210 5052 ts_tablet_manager.cc:902] T ec3891bfc4014b63afef6c259de00ca7 P 16440e365430405fb10366ab70e2b429: Initiating tablet copy from peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:29.711158 5052 tablet_copy_client.cc:308] T ec3891bfc4014b63afef6c259de00ca7 P 16440e365430405fb10366ab70e2b429: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.7:44241
I20240503 02:13:29.711927 457 tablet_copy_service.cc:140] P fbbc2be6382e47908ecb7c68c367bd26: Received BeginTabletCopySession request for tablet ec3891bfc4014b63afef6c259de00ca7 from peer 16440e365430405fb10366ab70e2b429 ({username='slave'} at 127.22.108.1:55517)
I20240503 02:13:29.712134 457 tablet_copy_service.cc:161] P fbbc2be6382e47908ecb7c68c367bd26: Beginning new tablet copy session on tablet ec3891bfc4014b63afef6c259de00ca7 from peer 16440e365430405fb10366ab70e2b429 at {username='slave'} at 127.22.108.1:55517: session id = 16440e365430405fb10366ab70e2b429-ec3891bfc4014b63afef6c259de00ca7
I20240503 02:13:29.714416 457 tablet_copy_source_session.cc:205] T ec3891bfc4014b63afef6c259de00ca7 P fbbc2be6382e47908ecb7c68c367bd26: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:29.715723 5052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec3891bfc4014b63afef6c259de00ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:29.719776 5052 tablet_copy_client.cc:783] T ec3891bfc4014b63afef6c259de00ca7 P 16440e365430405fb10366ab70e2b429: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:29.720024 5052 tablet_copy_client.cc:647] T ec3891bfc4014b63afef6c259de00ca7 P 16440e365430405fb10366ab70e2b429: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:29.721689 5052 tablet_copy_client.cc:523] T ec3891bfc4014b63afef6c259de00ca7 P 16440e365430405fb10366ab70e2b429: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:29.725006 5052 tablet_bootstrap.cc:492] T ec3891bfc4014b63afef6c259de00ca7 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:13:29.730263 5052 tablet_bootstrap.cc:492] T ec3891bfc4014b63afef6c259de00ca7 P 16440e365430405fb10366ab70e2b429: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:29.730675 5052 tablet_bootstrap.cc:492] T ec3891bfc4014b63afef6c259de00ca7 P 16440e365430405fb10366ab70e2b429: Bootstrap complete.
I20240503 02:13:29.730983 5052 ts_tablet_manager.cc:1367] T ec3891bfc4014b63afef6c259de00ca7 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20240503 02:13:29.731873 5052 raft_consensus.cc:348] T ec3891bfc4014b63afef6c259de00ca7 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:29.732177 5052 raft_consensus.cc:729] T ec3891bfc4014b63afef6c259de00ca7 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:13:29.732468 5052 consensus_queue.cc:260] T ec3891bfc4014b63afef6c259de00ca7 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:29.733363 5052 ts_tablet_manager.cc:1398] T ec3891bfc4014b63afef6c259de00ca7 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:13:29.734391 457 tablet_copy_service.cc:342] P fbbc2be6382e47908ecb7c68c367bd26: Request end of tablet copy session 16440e365430405fb10366ab70e2b429-ec3891bfc4014b63afef6c259de00ca7 received from {username='slave'} at 127.22.108.1:55517
I20240503 02:13:29.734552 457 tablet_copy_service.cc:433] P fbbc2be6382e47908ecb7c68c367bd26: ending tablet copy session 16440e365430405fb10366ab70e2b429-ec3891bfc4014b63afef6c259de00ca7 on tablet ec3891bfc4014b63afef6c259de00ca7 with peer 16440e365430405fb10366ab70e2b429
I20240503 02:13:29.734659 5087 ts_tablet_manager.cc:902] T 00aedc4902a745d6a859cbbf99529a7f P 155735c7f8b14997ac4c770aee2a9401: Initiating tablet copy from peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:29.739693 5089 ts_tablet_manager.cc:902] T f694fefb42d84318a6d91532105b3c90 P f575ce92c0224eeb80b9fae5f6561fbe: Initiating tablet copy from peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:29.745645 5087 tablet_copy_client.cc:308] T 00aedc4902a745d6a859cbbf99529a7f P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.6:39975
I20240503 02:13:29.746685 327 tablet_copy_service.cc:140] P 8a666e798fe8477d8dd1e51b385d5ead: Received BeginTabletCopySession request for tablet 00aedc4902a745d6a859cbbf99529a7f from peer 155735c7f8b14997ac4c770aee2a9401 ({username='slave'} at 127.22.108.2:53863)
I20240503 02:13:29.746940 327 tablet_copy_service.cc:161] P 8a666e798fe8477d8dd1e51b385d5ead: Beginning new tablet copy session on tablet 00aedc4902a745d6a859cbbf99529a7f from peer 155735c7f8b14997ac4c770aee2a9401 at {username='slave'} at 127.22.108.2:53863: session id = 155735c7f8b14997ac4c770aee2a9401-00aedc4902a745d6a859cbbf99529a7f
I20240503 02:13:29.752673 327 tablet_copy_source_session.cc:205] T 00aedc4902a745d6a859cbbf99529a7f P 8a666e798fe8477d8dd1e51b385d5ead: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:29.754998 5087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00aedc4902a745d6a859cbbf99529a7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:29.755044 5089 tablet_copy_client.cc:308] T f694fefb42d84318a6d91532105b3c90 P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.6:39975
I20240503 02:13:29.755676 327 tablet_copy_service.cc:140] P 8a666e798fe8477d8dd1e51b385d5ead: Received BeginTabletCopySession request for tablet f694fefb42d84318a6d91532105b3c90 from peer f575ce92c0224eeb80b9fae5f6561fbe ({username='slave'} at 127.22.108.8:34419)
I20240503 02:13:29.755822 327 tablet_copy_service.cc:161] P 8a666e798fe8477d8dd1e51b385d5ead: Beginning new tablet copy session on tablet f694fefb42d84318a6d91532105b3c90 from peer f575ce92c0224eeb80b9fae5f6561fbe at {username='slave'} at 127.22.108.8:34419: session id = f575ce92c0224eeb80b9fae5f6561fbe-f694fefb42d84318a6d91532105b3c90
I20240503 02:13:29.757895 327 tablet_copy_source_session.cc:205] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:29.759804 5087 tablet_copy_client.cc:783] T 00aedc4902a745d6a859cbbf99529a7f P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:29.760025 5087 tablet_copy_client.cc:647] T 00aedc4902a745d6a859cbbf99529a7f P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:29.762336 5087 tablet_copy_client.cc:523] T 00aedc4902a745d6a859cbbf99529a7f P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:29.762960 5089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f694fefb42d84318a6d91532105b3c90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:29.765115 5087 tablet_bootstrap.cc:492] T 00aedc4902a745d6a859cbbf99529a7f P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:13:29.768615 5089 tablet_copy_client.cc:783] T f694fefb42d84318a6d91532105b3c90 P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:29.768896 5089 tablet_copy_client.cc:647] T f694fefb42d84318a6d91532105b3c90 P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:29.770416 5089 tablet_copy_client.cc:523] T f694fefb42d84318a6d91532105b3c90 P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:29.771229 5087 tablet_bootstrap.cc:492] T 00aedc4902a745d6a859cbbf99529a7f P 155735c7f8b14997ac4c770aee2a9401: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:29.771651 5087 tablet_bootstrap.cc:492] T 00aedc4902a745d6a859cbbf99529a7f P 155735c7f8b14997ac4c770aee2a9401: Bootstrap complete.
I20240503 02:13:29.771871 5087 ts_tablet_manager.cc:1367] T 00aedc4902a745d6a859cbbf99529a7f P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.002s
I20240503 02:13:29.772495 5087 raft_consensus.cc:348] T 00aedc4902a745d6a859cbbf99529a7f P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:29.772814 5087 raft_consensus.cc:729] T 00aedc4902a745d6a859cbbf99529a7f P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:13:29.773061 5087 consensus_queue.cc:260] T 00aedc4902a745d6a859cbbf99529a7f P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:29.773713 5087 ts_tablet_manager.cc:1398] T 00aedc4902a745d6a859cbbf99529a7f P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:13:29.774564 327 tablet_copy_service.cc:342] P 8a666e798fe8477d8dd1e51b385d5ead: Request end of tablet copy session 155735c7f8b14997ac4c770aee2a9401-00aedc4902a745d6a859cbbf99529a7f received from {username='slave'} at 127.22.108.2:53863
I20240503 02:13:29.774739 327 tablet_copy_service.cc:433] P 8a666e798fe8477d8dd1e51b385d5ead: ending tablet copy session 155735c7f8b14997ac4c770aee2a9401-00aedc4902a745d6a859cbbf99529a7f on tablet 00aedc4902a745d6a859cbbf99529a7f with peer 155735c7f8b14997ac4c770aee2a9401
I20240503 02:13:29.775900 5089 tablet_bootstrap.cc:492] T f694fefb42d84318a6d91532105b3c90 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:29.782860 29874 consensus_queue.cc:237] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:29.783882 5089 tablet_bootstrap.cc:492] T f694fefb42d84318a6d91532105b3c90 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:29.784257 5089 tablet_bootstrap.cc:492] T f694fefb42d84318a6d91532105b3c90 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap complete.
I20240503 02:13:29.784504 5089 ts_tablet_manager.cc:1367] T f694fefb42d84318a6d91532105b3c90 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.004s
I20240503 02:13:29.785859 30264 raft_consensus.cc:1261] T 89938d4c793a44408795398078cc2ee0 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:29.785864 30004 raft_consensus.cc:1261] T 89938d4c793a44408795398078cc2ee0 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:29.786563 5089 raft_consensus.cc:348] T f694fefb42d84318a6d91532105b3c90 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:29.786629 5065 consensus_queue.cc:1035] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:29.786790 5089 raft_consensus.cc:729] T f694fefb42d84318a6d91532105b3c90 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:29.787070 5089 consensus_queue.cc:260] T f694fefb42d84318a6d91532105b3c90 P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
W20240503 02:13:29.787158 29809 consensus_peers.cc:485] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429 -> Peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): Couldn't send request to peer 8a666e798fe8477d8dd1e51b385d5ead. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 89938d4c793a44408795398078cc2ee0. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:29.787457 5066 consensus_queue.cc:1035] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:29.795715 5089 ts_tablet_manager.cc:1398] T f694fefb42d84318a6d91532105b3c90 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.011s user 0.002s sys 0.002s
I20240503 02:13:29.797703 327 tablet_copy_service.cc:342] P 8a666e798fe8477d8dd1e51b385d5ead: Request end of tablet copy session f575ce92c0224eeb80b9fae5f6561fbe-f694fefb42d84318a6d91532105b3c90 received from {username='slave'} at 127.22.108.8:34419
I20240503 02:13:29.797864 327 tablet_copy_service.cc:433] P 8a666e798fe8477d8dd1e51b385d5ead: ending tablet copy session f575ce92c0224eeb80b9fae5f6561fbe-f694fefb42d84318a6d91532105b3c90 on tablet f694fefb42d84318a6d91532105b3c90 with peer f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:13:29.804462 5065 raft_consensus.cc:2932] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:29.805246 30264 raft_consensus.cc:2932] T 89938d4c793a44408795398078cc2ee0 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:29.810096 30004 raft_consensus.cc:2932] T 89938d4c793a44408795398078cc2ee0 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:29.812927 29725 catalog_manager.cc:5484] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429 reported cstate change: config changed from index -1 to 2, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:29.873111 5024 ts_tablet_manager.cc:902] T 04e47da8cf7445dcacaf1f7c8950bd34 P fbbc2be6382e47908ecb7c68c367bd26: Initiating tablet copy from peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:13:29.873548 29874 consensus_queue.cc:237] T 9102ab9545c54e82b642673795cb747a P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:29.876597 433 raft_consensus.cc:1261] T 9102ab9545c54e82b642673795cb747a P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:29.877235 5066 consensus_queue.cc:1035] T 9102ab9545c54e82b642673795cb747a P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:13:29.878043 29808 consensus_peers.cc:485] T 9102ab9545c54e82b642673795cb747a P 16440e365430405fb10366ab70e2b429 -> Peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Couldn't send request to peer a9094240860e43bfb129193f3bfd5be6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9102ab9545c54e82b642673795cb747a. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:29.879211 30004 raft_consensus.cc:1261] T 9102ab9545c54e82b642673795cb747a P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:29.879791 5065 consensus_queue.cc:1035] T 9102ab9545c54e82b642673795cb747a P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:29.884514 5066 raft_consensus.cc:2932] T 9102ab9545c54e82b642673795cb747a P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:29.888317 30004 raft_consensus.cc:2932] T 9102ab9545c54e82b642673795cb747a P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:29.896154 434 raft_consensus.cc:2932] T 9102ab9545c54e82b642673795cb747a P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:29.897323 29729 catalog_manager.cc:5484] T 9102ab9545c54e82b642673795cb747a P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:29.934569 5024 tablet_copy_client.cc:308] T 04e47da8cf7445dcacaf1f7c8950bd34 P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.2:37873
I20240503 02:13:29.935200 30026 tablet_copy_service.cc:140] P 155735c7f8b14997ac4c770aee2a9401: Received BeginTabletCopySession request for tablet 04e47da8cf7445dcacaf1f7c8950bd34 from peer fbbc2be6382e47908ecb7c68c367bd26 ({username='slave'} at 127.22.108.7:60369)
I20240503 02:13:29.935359 30026 tablet_copy_service.cc:161] P 155735c7f8b14997ac4c770aee2a9401: Beginning new tablet copy session on tablet 04e47da8cf7445dcacaf1f7c8950bd34 from peer fbbc2be6382e47908ecb7c68c367bd26 at {username='slave'} at 127.22.108.7:60369: session id = fbbc2be6382e47908ecb7c68c367bd26-04e47da8cf7445dcacaf1f7c8950bd34
I20240503 02:13:29.938416 30026 tablet_copy_source_session.cc:205] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:29.939797 5024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04e47da8cf7445dcacaf1f7c8950bd34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:29.945673 5024 tablet_copy_client.cc:783] T 04e47da8cf7445dcacaf1f7c8950bd34 P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:29.945935 5024 tablet_copy_client.cc:647] T 04e47da8cf7445dcacaf1f7c8950bd34 P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:29.948736 434 raft_consensus.cc:1203] T 63b416f4e004490b8e38a96fd065dfea P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:29.950215 5024 tablet_copy_client.cc:523] T 04e47da8cf7445dcacaf1f7c8950bd34 P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:29.953575 5024 tablet_bootstrap.cc:492] T 04e47da8cf7445dcacaf1f7c8950bd34 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:30.026961 5024 tablet_bootstrap.cc:492] T 04e47da8cf7445dcacaf1f7c8950bd34 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:30.027743 5024 tablet_bootstrap.cc:492] T 04e47da8cf7445dcacaf1f7c8950bd34 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap complete.
I20240503 02:13:30.028276 5024 ts_tablet_manager.cc:1367] T 04e47da8cf7445dcacaf1f7c8950bd34 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.075s user 0.047s sys 0.004s
I20240503 02:13:30.029613 5024 raft_consensus.cc:348] T 04e47da8cf7445dcacaf1f7c8950bd34 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:30.029978 5024 raft_consensus.cc:729] T 04e47da8cf7445dcacaf1f7c8950bd34 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:30.030308 5024 consensus_queue.cc:260] T 04e47da8cf7445dcacaf1f7c8950bd34 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:30.031054 5024 ts_tablet_manager.cc:1398] T 04e47da8cf7445dcacaf1f7c8950bd34 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:30.032264 30026 tablet_copy_service.cc:342] P 155735c7f8b14997ac4c770aee2a9401: Request end of tablet copy session fbbc2be6382e47908ecb7c68c367bd26-04e47da8cf7445dcacaf1f7c8950bd34 received from {username='slave'} at 127.22.108.7:60369
I20240503 02:13:30.032382 30026 tablet_copy_service.cc:433] P 155735c7f8b14997ac4c770aee2a9401: ending tablet copy session fbbc2be6382e47908ecb7c68c367bd26-04e47da8cf7445dcacaf1f7c8950bd34 on tablet 04e47da8cf7445dcacaf1f7c8950bd34 with peer fbbc2be6382e47908ecb7c68c367bd26
I20240503 02:13:30.042913 434 consensus_queue.cc:237] T 83e8dad806564b128d5f0fe559d8ffe1 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:30.045431 30135 raft_consensus.cc:1261] T 83e8dad806564b128d5f0fe559d8ffe1 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:13:30.045743 371 consensus_peers.cc:485] T 83e8dad806564b128d5f0fe559d8ffe1 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Couldn't send request to peer 155735c7f8b14997ac4c770aee2a9401. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 83e8dad806564b128d5f0fe559d8ffe1. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:30.046015 4830 consensus_queue.cc:1035] T 83e8dad806564b128d5f0fe559d8ffe1 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:30.047104 32641 raft_consensus.cc:1261] T 83e8dad806564b128d5f0fe559d8ffe1 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:30.048195 4830 consensus_queue.cc:1035] T 83e8dad806564b128d5f0fe559d8ffe1 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:30.052526 32638 raft_consensus.cc:2932] T 83e8dad806564b128d5f0fe559d8ffe1 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:30.051970 5149 raft_consensus.cc:2932] T 83e8dad806564b128d5f0fe559d8ffe1 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:30.058022 29729 catalog_manager.cc:5484] T 83e8dad806564b128d5f0fe559d8ffe1 P 9a4d92bc25374fb3960fde7d10be309d reported cstate change: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:30.060395 30135 raft_consensus.cc:2932] T 83e8dad806564b128d5f0fe559d8ffe1 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:30.073248 30264 raft_consensus.cc:1203] T 1758ec3783b24d3c820ed0d6dc118e69 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:30.102416 30135 raft_consensus.cc:1203] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:30.102739 5089 ts_tablet_manager.cc:902] T 5040203bd48549ff9c8f64c71d6bc598 P f575ce92c0224eeb80b9fae5f6561fbe: Initiating tablet copy from peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:13:30.108392 5089 tablet_copy_client.cc:308] T 5040203bd48549ff9c8f64c71d6bc598 P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.3:42679
I20240503 02:13:30.108321 305 consensus_queue.cc:237] T e165c71f8b6645a6936b58b2a8da1842 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:30.109730 30156 tablet_copy_service.cc:140] P a9094240860e43bfb129193f3bfd5be6: Received BeginTabletCopySession request for tablet 5040203bd48549ff9c8f64c71d6bc598 from peer f575ce92c0224eeb80b9fae5f6561fbe ({username='slave'} at 127.22.108.8:37273)
I20240503 02:13:30.109900 30156 tablet_copy_service.cc:161] P a9094240860e43bfb129193f3bfd5be6: Beginning new tablet copy session on tablet 5040203bd48549ff9c8f64c71d6bc598 from peer f575ce92c0224eeb80b9fae5f6561fbe at {username='slave'} at 127.22.108.8:37273: session id = f575ce92c0224eeb80b9fae5f6561fbe-5040203bd48549ff9c8f64c71d6bc598
I20240503 02:13:30.112218 30156 tablet_copy_source_session.cc:205] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:30.113508 5089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5040203bd48549ff9c8f64c71d6bc598. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:13:30.114212 32709 consensus_peers.cc:485] T e165c71f8b6645a6936b58b2a8da1842 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): Couldn't send request to peer 1f10f80f13c84e80b03eda7085a3b012. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e165c71f8b6645a6936b58b2a8da1842. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:30.115027 30135 raft_consensus.cc:1261] T e165c71f8b6645a6936b58b2a8da1842 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:30.115456 4985 consensus_queue.cc:1035] T e165c71f8b6645a6936b58b2a8da1842 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:30.117309 566 raft_consensus.cc:1261] T e165c71f8b6645a6936b58b2a8da1842 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:30.117834 4985 consensus_queue.cc:1035] T e165c71f8b6645a6936b58b2a8da1842 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:30.118006 566 raft_consensus.cc:1203] T f694fefb42d84318a6d91532105b3c90 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:30.122795 5089 tablet_copy_client.cc:783] T 5040203bd48549ff9c8f64c71d6bc598 P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:30.123224 5089 tablet_copy_client.cc:647] T 5040203bd48549ff9c8f64c71d6bc598 P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:30.122960 4985 raft_consensus.cc:2932] T e165c71f8b6645a6936b58b2a8da1842 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:30.124842 566 raft_consensus.cc:2932] T e165c71f8b6645a6936b58b2a8da1842 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:30.128031 5089 tablet_copy_client.cc:523] T 5040203bd48549ff9c8f64c71d6bc598 P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:30.132546 5089 tablet_bootstrap.cc:492] T 5040203bd48549ff9c8f64c71d6bc598 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:30.135483 29729 catalog_manager.cc:5484] T e165c71f8b6645a6936b58b2a8da1842 P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:30.148669 30135 raft_consensus.cc:2932] T e165c71f8b6645a6936b58b2a8da1842 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:30.180706 29874 raft_consensus.cc:1203] T ec3891bfc4014b63afef6c259de00ca7 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:30.181385 29872 raft_consensus.cc:1203] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:30.205235 5185 ts_tablet_manager.cc:902] T 89938d4c793a44408795398078cc2ee0 P 8a666e798fe8477d8dd1e51b385d5ead: Initiating tablet copy from peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:13:30.224383 30005 raft_consensus.cc:1203] T 00aedc4902a745d6a859cbbf99529a7f P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:30.248116 32638 consensus_queue.cc:237] T 16cf978ac35341d4adea71a17a91194c P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:30.252249 305 raft_consensus.cc:1261] T 16cf978ac35341d4adea71a17a91194c P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:30.252805 4915 consensus_queue.cc:1035] T 16cf978ac35341d4adea71a17a91194c P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:30.253316 434 raft_consensus.cc:1261] T 16cf978ac35341d4adea71a17a91194c P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:13:30.253804 32576 consensus_peers.cc:485] T 16cf978ac35341d4adea71a17a91194c P 9a4d92bc25374fb3960fde7d10be309d -> Peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Couldn't send request to peer 16440e365430405fb10366ab70e2b429. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 16cf978ac35341d4adea71a17a91194c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:30.256557 4915 consensus_queue.cc:1035] T 16cf978ac35341d4adea71a17a91194c P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:30.262995 5185 tablet_copy_client.cc:308] T 89938d4c793a44408795398078cc2ee0 P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.1:36719
I20240503 02:13:30.263315 5089 tablet_bootstrap.cc:492] T 5040203bd48549ff9c8f64c71d6bc598 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:30.263629 29895 tablet_copy_service.cc:140] P 16440e365430405fb10366ab70e2b429: Received BeginTabletCopySession request for tablet 89938d4c793a44408795398078cc2ee0 from peer 8a666e798fe8477d8dd1e51b385d5ead ({username='slave'} at 127.22.108.6:49099)
I20240503 02:13:30.263833 29895 tablet_copy_service.cc:161] P 16440e365430405fb10366ab70e2b429: Beginning new tablet copy session on tablet 89938d4c793a44408795398078cc2ee0 from peer 8a666e798fe8477d8dd1e51b385d5ead at {username='slave'} at 127.22.108.6:49099: session id = 8a666e798fe8477d8dd1e51b385d5ead-89938d4c793a44408795398078cc2ee0
I20240503 02:13:30.263785 5089 tablet_bootstrap.cc:492] T 5040203bd48549ff9c8f64c71d6bc598 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap complete.
I20240503 02:13:30.264467 5089 ts_tablet_manager.cc:1367] T 5040203bd48549ff9c8f64c71d6bc598 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.132s user 0.076s sys 0.010s
I20240503 02:13:30.264969 4915 raft_consensus.cc:2932] T 16cf978ac35341d4adea71a17a91194c P 9a4d92bc25374fb3960fde7d10be309d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:30.265792 5089 raft_consensus.cc:348] T 5040203bd48549ff9c8f64c71d6bc598 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:30.265970 5089 raft_consensus.cc:729] T 5040203bd48549ff9c8f64c71d6bc598 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:30.266170 5089 consensus_queue.cc:260] T 5040203bd48549ff9c8f64c71d6bc598 P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:30.266623 5089 ts_tablet_manager.cc:1398] T 5040203bd48549ff9c8f64c71d6bc598 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:13:30.266660 29895 tablet_copy_source_session.cc:205] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:30.267302 433 raft_consensus.cc:2932] T 16cf978ac35341d4adea71a17a91194c P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:30.268543 30156 tablet_copy_service.cc:342] P a9094240860e43bfb129193f3bfd5be6: Request end of tablet copy session f575ce92c0224eeb80b9fae5f6561fbe-5040203bd48549ff9c8f64c71d6bc598 received from {username='slave'} at 127.22.108.8:37273
I20240503 02:13:30.268673 30156 tablet_copy_service.cc:433] P a9094240860e43bfb129193f3bfd5be6: ending tablet copy session f575ce92c0224eeb80b9fae5f6561fbe-5040203bd48549ff9c8f64c71d6bc598 on tablet 5040203bd48549ff9c8f64c71d6bc598 with peer f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:13:30.272392 5185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89938d4c793a44408795398078cc2ee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:30.275755 305 raft_consensus.cc:2932] T 16cf978ac35341d4adea71a17a91194c P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:30.279021 29726 catalog_manager.cc:5484] T 16cf978ac35341d4adea71a17a91194c P 9a4d92bc25374fb3960fde7d10be309d reported cstate change: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New cstate: current_term: 1 leader_uuid: "9a4d92bc25374fb3960fde7d10be309d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:30.283772 5185 tablet_copy_client.cc:783] T 89938d4c793a44408795398078cc2ee0 P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:30.284107 5185 tablet_copy_client.cc:647] T 89938d4c793a44408795398078cc2ee0 P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:30.288653 5185 tablet_copy_client.cc:523] T 89938d4c793a44408795398078cc2ee0 P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:30.291582 5185 tablet_bootstrap.cc:492] T 89938d4c793a44408795398078cc2ee0 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:30.292985 433 raft_consensus.cc:1203] T 04e47da8cf7445dcacaf1f7c8950bd34 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:30.300609 5185 tablet_bootstrap.cc:492] T 89938d4c793a44408795398078cc2ee0 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:30.301724 5185 tablet_bootstrap.cc:492] T 89938d4c793a44408795398078cc2ee0 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap complete.
I20240503 02:13:30.302008 5185 ts_tablet_manager.cc:1367] T 89938d4c793a44408795398078cc2ee0 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240503 02:13:30.302841 5185 raft_consensus.cc:348] T 89938d4c793a44408795398078cc2ee0 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:30.303078 5185 raft_consensus.cc:729] T 89938d4c793a44408795398078cc2ee0 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:30.303519 5185 consensus_queue.cc:260] T 89938d4c793a44408795398078cc2ee0 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:30.304563 5185 ts_tablet_manager.cc:1398] T 89938d4c793a44408795398078cc2ee0 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:30.305686 29895 tablet_copy_service.cc:342] P 16440e365430405fb10366ab70e2b429: Request end of tablet copy session 8a666e798fe8477d8dd1e51b385d5ead-89938d4c793a44408795398078cc2ee0 received from {username='slave'} at 127.22.108.6:49099
I20240503 02:13:30.305825 29895 tablet_copy_service.cc:433] P 16440e365430405fb10366ab70e2b429: ending tablet copy session 8a666e798fe8477d8dd1e51b385d5ead-89938d4c793a44408795398078cc2ee0 on tablet 89938d4c793a44408795398078cc2ee0 with peer 8a666e798fe8477d8dd1e51b385d5ead
I20240503 02:13:30.335354 30264 consensus_queue.cc:237] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:30.337767 32641 raft_consensus.cc:1261] T 9fa3f1e452c24c7d83a36f10284b8b5d P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:30.338740 5222 consensus_queue.cc:1035] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:30.338981 433 raft_consensus.cc:1261] T 9fa3f1e452c24c7d83a36f10284b8b5d P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:30.339581 5184 consensus_queue.cc:1035] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:13:30.339931 30199 consensus_peers.cc:485] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 -> Peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Couldn't send request to peer a9094240860e43bfb129193f3bfd5be6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9fa3f1e452c24c7d83a36f10284b8b5d. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:30.346545 5184 raft_consensus.cc:2932] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:30.349618 434 raft_consensus.cc:2932] T 9fa3f1e452c24c7d83a36f10284b8b5d P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:30.356704 29725 catalog_manager.cc:5484] T 9fa3f1e452c24c7d83a36f10284b8b5d P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:30.383034 32638 raft_consensus.cc:2932] T 9fa3f1e452c24c7d83a36f10284b8b5d P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:30.408754 5232 ts_tablet_manager.cc:902] T 9102ab9545c54e82b642673795cb747a P a9094240860e43bfb129193f3bfd5be6: Initiating tablet copy from peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:13:30.457965 5241 ts_tablet_manager.cc:902] T 83e8dad806564b128d5f0fe559d8ffe1 P 155735c7f8b14997ac4c770aee2a9401: Initiating tablet copy from peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:30.473459 5232 tablet_copy_client.cc:308] T 9102ab9545c54e82b642673795cb747a P a9094240860e43bfb129193f3bfd5be6: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.1:36719
I20240503 02:13:30.474478 29895 tablet_copy_service.cc:140] P 16440e365430405fb10366ab70e2b429: Received BeginTabletCopySession request for tablet 9102ab9545c54e82b642673795cb747a from peer a9094240860e43bfb129193f3bfd5be6 ({username='slave'} at 127.22.108.3:49185)
I20240503 02:13:30.474731 29895 tablet_copy_service.cc:161] P 16440e365430405fb10366ab70e2b429: Beginning new tablet copy session on tablet 9102ab9545c54e82b642673795cb747a from peer a9094240860e43bfb129193f3bfd5be6 at {username='slave'} at 127.22.108.3:49185: session id = a9094240860e43bfb129193f3bfd5be6-9102ab9545c54e82b642673795cb747a
I20240503 02:13:30.480619 29895 tablet_copy_source_session.cc:205] T 9102ab9545c54e82b642673795cb747a P 16440e365430405fb10366ab70e2b429: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:30.482082 5232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9102ab9545c54e82b642673795cb747a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:30.490473 5232 tablet_copy_client.cc:783] T 9102ab9545c54e82b642673795cb747a P a9094240860e43bfb129193f3bfd5be6: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:30.490744 5232 tablet_copy_client.cc:647] T 9102ab9545c54e82b642673795cb747a P a9094240860e43bfb129193f3bfd5be6: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:30.494827 5232 tablet_copy_client.cc:523] T 9102ab9545c54e82b642673795cb747a P a9094240860e43bfb129193f3bfd5be6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:30.498239 5232 tablet_bootstrap.cc:492] T 9102ab9545c54e82b642673795cb747a P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:13:30.541972 5232 tablet_bootstrap.cc:492] T 9102ab9545c54e82b642673795cb747a P a9094240860e43bfb129193f3bfd5be6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:30.542479 5232 tablet_bootstrap.cc:492] T 9102ab9545c54e82b642673795cb747a P a9094240860e43bfb129193f3bfd5be6: Bootstrap complete.
I20240503 02:13:30.542788 5232 ts_tablet_manager.cc:1367] T 9102ab9545c54e82b642673795cb747a P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.045s user 0.021s sys 0.017s
I20240503 02:13:30.543516 5232 raft_consensus.cc:348] T 9102ab9545c54e82b642673795cb747a P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:30.543829 5232 raft_consensus.cc:729] T 9102ab9545c54e82b642673795cb747a P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:13:30.544144 5232 consensus_queue.cc:260] T 9102ab9545c54e82b642673795cb747a P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:30.544760 5232 ts_tablet_manager.cc:1398] T 9102ab9545c54e82b642673795cb747a P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:13:30.547174 29895 tablet_copy_service.cc:342] P 16440e365430405fb10366ab70e2b429: Request end of tablet copy session a9094240860e43bfb129193f3bfd5be6-9102ab9545c54e82b642673795cb747a received from {username='slave'} at 127.22.108.3:49185
I20240503 02:13:30.547348 29895 tablet_copy_service.cc:433] P 16440e365430405fb10366ab70e2b429: ending tablet copy session a9094240860e43bfb129193f3bfd5be6-9102ab9545c54e82b642673795cb747a on tablet 9102ab9545c54e82b642673795cb747a with peer a9094240860e43bfb129193f3bfd5be6
I20240503 02:13:30.560164 5241 tablet_copy_client.cc:308] T 83e8dad806564b128d5f0fe559d8ffe1 P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.7:44241
I20240503 02:13:30.560981 457 tablet_copy_service.cc:140] P fbbc2be6382e47908ecb7c68c367bd26: Received BeginTabletCopySession request for tablet 83e8dad806564b128d5f0fe559d8ffe1 from peer 155735c7f8b14997ac4c770aee2a9401 ({username='slave'} at 127.22.108.2:48617)
I20240503 02:13:30.561200 457 tablet_copy_service.cc:161] P fbbc2be6382e47908ecb7c68c367bd26: Beginning new tablet copy session on tablet 83e8dad806564b128d5f0fe559d8ffe1 from peer 155735c7f8b14997ac4c770aee2a9401 at {username='slave'} at 127.22.108.2:48617: session id = 155735c7f8b14997ac4c770aee2a9401-83e8dad806564b128d5f0fe559d8ffe1
I20240503 02:13:30.564145 457 tablet_copy_source_session.cc:205] T 83e8dad806564b128d5f0fe559d8ffe1 P fbbc2be6382e47908ecb7c68c367bd26: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:30.565960 5241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83e8dad806564b128d5f0fe559d8ffe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:30.572166 5241 tablet_copy_client.cc:783] T 83e8dad806564b128d5f0fe559d8ffe1 P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:30.572399 5241 tablet_copy_client.cc:647] T 83e8dad806564b128d5f0fe559d8ffe1 P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:30.575776 5241 tablet_copy_client.cc:523] T 83e8dad806564b128d5f0fe559d8ffe1 P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:30.580256 5241 tablet_bootstrap.cc:492] T 83e8dad806564b128d5f0fe559d8ffe1 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:13:30.589771 5250 ts_tablet_manager.cc:902] T e165c71f8b6645a6936b58b2a8da1842 P 1f10f80f13c84e80b03eda7085a3b012: Initiating tablet copy from peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:30.593791 5241 tablet_bootstrap.cc:492] T 83e8dad806564b128d5f0fe559d8ffe1 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:30.594288 5241 tablet_bootstrap.cc:492] T 83e8dad806564b128d5f0fe559d8ffe1 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap complete.
I20240503 02:13:30.594599 5241 ts_tablet_manager.cc:1367] T 83e8dad806564b128d5f0fe559d8ffe1 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.007s
I20240503 02:13:30.595384 5241 raft_consensus.cc:348] T 83e8dad806564b128d5f0fe559d8ffe1 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:30.595672 5241 raft_consensus.cc:729] T 83e8dad806564b128d5f0fe559d8ffe1 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:13:30.595911 5241 consensus_queue.cc:260] T 83e8dad806564b128d5f0fe559d8ffe1 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:30.596490 5241 ts_tablet_manager.cc:1398] T 83e8dad806564b128d5f0fe559d8ffe1 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:13:30.597827 457 tablet_copy_service.cc:342] P fbbc2be6382e47908ecb7c68c367bd26: Request end of tablet copy session 155735c7f8b14997ac4c770aee2a9401-83e8dad806564b128d5f0fe559d8ffe1 received from {username='slave'} at 127.22.108.2:48617
I20240503 02:13:30.597976 457 tablet_copy_service.cc:433] P fbbc2be6382e47908ecb7c68c367bd26: ending tablet copy session 155735c7f8b14997ac4c770aee2a9401-83e8dad806564b128d5f0fe559d8ffe1 on tablet 83e8dad806564b128d5f0fe559d8ffe1 with peer 155735c7f8b14997ac4c770aee2a9401
I20240503 02:13:30.600970 5250 tablet_copy_client.cc:308] T e165c71f8b6645a6936b58b2a8da1842 P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.6:39975
I20240503 02:13:30.605994 327 tablet_copy_service.cc:140] P 8a666e798fe8477d8dd1e51b385d5ead: Received BeginTabletCopySession request for tablet e165c71f8b6645a6936b58b2a8da1842 from peer 1f10f80f13c84e80b03eda7085a3b012 ({username='slave'} at 127.22.108.4:53319)
I20240503 02:13:30.606176 327 tablet_copy_service.cc:161] P 8a666e798fe8477d8dd1e51b385d5ead: Beginning new tablet copy session on tablet e165c71f8b6645a6936b58b2a8da1842 from peer 1f10f80f13c84e80b03eda7085a3b012 at {username='slave'} at 127.22.108.4:53319: session id = 1f10f80f13c84e80b03eda7085a3b012-e165c71f8b6645a6936b58b2a8da1842
I20240503 02:13:30.609163 327 tablet_copy_source_session.cc:205] T e165c71f8b6645a6936b58b2a8da1842 P 8a666e798fe8477d8dd1e51b385d5ead: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:30.611063 5250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e165c71f8b6645a6936b58b2a8da1842. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:30.615417 5250 tablet_copy_client.cc:783] T e165c71f8b6645a6936b58b2a8da1842 P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:30.615620 5250 tablet_copy_client.cc:647] T e165c71f8b6645a6936b58b2a8da1842 P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:30.617427 5250 tablet_copy_client.cc:523] T e165c71f8b6645a6936b58b2a8da1842 P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:30.620131 5250 tablet_bootstrap.cc:492] T e165c71f8b6645a6936b58b2a8da1842 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:13:30.626086 5250 tablet_bootstrap.cc:492] T e165c71f8b6645a6936b58b2a8da1842 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:30.626447 5250 tablet_bootstrap.cc:492] T e165c71f8b6645a6936b58b2a8da1842 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap complete.
I20240503 02:13:30.626711 5250 ts_tablet_manager.cc:1367] T e165c71f8b6645a6936b58b2a8da1842 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.001s
I20240503 02:13:30.627383 5250 raft_consensus.cc:348] T e165c71f8b6645a6936b58b2a8da1842 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:30.627609 5250 raft_consensus.cc:729] T e165c71f8b6645a6936b58b2a8da1842 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:13:30.627838 5250 consensus_queue.cc:260] T e165c71f8b6645a6936b58b2a8da1842 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:30.628355 5250 ts_tablet_manager.cc:1398] T e165c71f8b6645a6936b58b2a8da1842 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:13:30.629280 327 tablet_copy_service.cc:342] P 8a666e798fe8477d8dd1e51b385d5ead: Request end of tablet copy session 1f10f80f13c84e80b03eda7085a3b012-e165c71f8b6645a6936b58b2a8da1842 received from {username='slave'} at 127.22.108.4:53319
I20240503 02:13:30.629451 327 tablet_copy_service.cc:433] P 8a666e798fe8477d8dd1e51b385d5ead: ending tablet copy session 1f10f80f13c84e80b03eda7085a3b012-e165c71f8b6645a6936b58b2a8da1842 on tablet e165c71f8b6645a6936b58b2a8da1842 with peer 1f10f80f13c84e80b03eda7085a3b012
I20240503 02:13:30.674374 306 raft_consensus.cc:1203] T 89938d4c793a44408795398078cc2ee0 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:30.721817 5286 ts_tablet_manager.cc:902] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429: Initiating tablet copy from peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:13:30.723963 565 raft_consensus.cc:1203] T 5040203bd48549ff9c8f64c71d6bc598 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:30.741873 5286 tablet_copy_client.cc:308] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.5:35595
I20240503 02:13:30.742667 32662 tablet_copy_service.cc:140] P 9a4d92bc25374fb3960fde7d10be309d: Received BeginTabletCopySession request for tablet 16cf978ac35341d4adea71a17a91194c from peer 16440e365430405fb10366ab70e2b429 ({username='slave'} at 127.22.108.1:43971)
I20240503 02:13:30.742871 32662 tablet_copy_service.cc:161] P 9a4d92bc25374fb3960fde7d10be309d: Beginning new tablet copy session on tablet 16cf978ac35341d4adea71a17a91194c from peer 16440e365430405fb10366ab70e2b429 at {username='slave'} at 127.22.108.1:43971: session id = 16440e365430405fb10366ab70e2b429-16cf978ac35341d4adea71a17a91194c
I20240503 02:13:30.745829 32662 tablet_copy_source_session.cc:205] T 16cf978ac35341d4adea71a17a91194c P 9a4d92bc25374fb3960fde7d10be309d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:30.749466 5286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16cf978ac35341d4adea71a17a91194c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:30.756516 5286 tablet_copy_client.cc:783] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:30.756832 5286 tablet_copy_client.cc:647] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:30.758620 5286 tablet_copy_client.cc:523] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:30.763654 5286 tablet_bootstrap.cc:492] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:13:30.777732 5286 tablet_bootstrap.cc:492] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:30.778525 5286 tablet_bootstrap.cc:492] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429: Bootstrap complete.
I20240503 02:13:30.778934 5286 ts_tablet_manager.cc:1367] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20240503 02:13:30.781908 5286 raft_consensus.cc:348] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:30.782339 5286 raft_consensus.cc:729] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:13:30.782686 5286 consensus_queue.cc:260] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:30.783679 5286 ts_tablet_manager.cc:1398] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:13:30.785020 32662 tablet_copy_service.cc:342] P 9a4d92bc25374fb3960fde7d10be309d: Request end of tablet copy session 16440e365430405fb10366ab70e2b429-16cf978ac35341d4adea71a17a91194c received from {username='slave'} at 127.22.108.1:43971
I20240503 02:13:30.785203 32662 tablet_copy_service.cc:433] P 9a4d92bc25374fb3960fde7d10be309d: ending tablet copy session 16440e365430405fb10366ab70e2b429-16cf978ac35341d4adea71a17a91194c on tablet 16cf978ac35341d4adea71a17a91194c with peer 16440e365430405fb10366ab70e2b429
I20240503 02:13:30.833287 32755 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:30.850013 29853 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:30.912256 30244 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:30.913796 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:30.918774 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:30.935480 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:30.940878 5232 ts_tablet_manager.cc:902] T 9fa3f1e452c24c7d83a36f10284b8b5d P a9094240860e43bfb129193f3bfd5be6: Initiating tablet copy from peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:13:30.950871 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:30.965297 30132 raft_consensus.cc:1203] T 9102ab9545c54e82b642673795cb747a P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:30.976171 30005 raft_consensus.cc:1203] T 83e8dad806564b128d5f0fe559d8ffe1 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:31.040915 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:31.045976 5232 tablet_copy_client.cc:308] T 9fa3f1e452c24c7d83a36f10284b8b5d P a9094240860e43bfb129193f3bfd5be6: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.4:33935
I20240503 02:13:31.046705 30286 tablet_copy_service.cc:140] P 1f10f80f13c84e80b03eda7085a3b012: Received BeginTabletCopySession request for tablet 9fa3f1e452c24c7d83a36f10284b8b5d from peer a9094240860e43bfb129193f3bfd5be6 ({username='slave'} at 127.22.108.3:41281)
I20240503 02:13:31.046861 30286 tablet_copy_service.cc:161] P 1f10f80f13c84e80b03eda7085a3b012: Beginning new tablet copy session on tablet 9fa3f1e452c24c7d83a36f10284b8b5d from peer a9094240860e43bfb129193f3bfd5be6 at {username='slave'} at 127.22.108.3:41281: session id = a9094240860e43bfb129193f3bfd5be6-9fa3f1e452c24c7d83a36f10284b8b5d
I20240503 02:13:31.053472 30286 tablet_copy_source_session.cc:205] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:31.055406 5232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fa3f1e452c24c7d83a36f10284b8b5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:31.069859 5232 tablet_copy_client.cc:783] T 9fa3f1e452c24c7d83a36f10284b8b5d P a9094240860e43bfb129193f3bfd5be6: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:31.070133 5232 tablet_copy_client.cc:647] T 9fa3f1e452c24c7d83a36f10284b8b5d P a9094240860e43bfb129193f3bfd5be6: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:31.099644 5232 tablet_copy_client.cc:523] T 9fa3f1e452c24c7d83a36f10284b8b5d P a9094240860e43bfb129193f3bfd5be6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:31.103168 5232 tablet_bootstrap.cc:492] T 9fa3f1e452c24c7d83a36f10284b8b5d P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:13:31.117815 5232 tablet_bootstrap.cc:492] T 9fa3f1e452c24c7d83a36f10284b8b5d P a9094240860e43bfb129193f3bfd5be6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:31.118142 5232 tablet_bootstrap.cc:492] T 9fa3f1e452c24c7d83a36f10284b8b5d P a9094240860e43bfb129193f3bfd5be6: Bootstrap complete.
I20240503 02:13:31.118366 5232 ts_tablet_manager.cc:1367] T 9fa3f1e452c24c7d83a36f10284b8b5d P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.001s
I20240503 02:13:31.118897 5232 raft_consensus.cc:348] T 9fa3f1e452c24c7d83a36f10284b8b5d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:31.119058 5232 raft_consensus.cc:729] T 9fa3f1e452c24c7d83a36f10284b8b5d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:13:31.119292 5232 consensus_queue.cc:260] T 9fa3f1e452c24c7d83a36f10284b8b5d P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:31.121177 5232 ts_tablet_manager.cc:1398] T 9fa3f1e452c24c7d83a36f10284b8b5d P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:13:31.123049 30286 tablet_copy_service.cc:342] P 1f10f80f13c84e80b03eda7085a3b012: Request end of tablet copy session a9094240860e43bfb129193f3bfd5be6-9fa3f1e452c24c7d83a36f10284b8b5d received from {username='slave'} at 127.22.108.3:41281
I20240503 02:13:31.123150 30286 tablet_copy_service.cc:433] P 1f10f80f13c84e80b03eda7085a3b012: ending tablet copy session a9094240860e43bfb129193f3bfd5be6-9fa3f1e452c24c7d83a36f10284b8b5d on tablet 9fa3f1e452c24c7d83a36f10284b8b5d with peer a9094240860e43bfb129193f3bfd5be6
I20240503 02:13:31.135757 29872 raft_consensus.cc:1203] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:31.196880 30266 raft_consensus.cc:1203] T e165c71f8b6645a6936b58b2a8da1842 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240503 02:13:31.435499 30200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.6:39975, user_credentials={real_user=slave}} blocked reactor thread for 56895.2us
I20240503 02:13:31.437695 30135 raft_consensus.cc:1203] T 9fa3f1e452c24c7d83a36f10284b8b5d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:31.486583 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:31.529067 29853 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:31.567620 32755 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:31.572292 30244 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:31.622372 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:31.628353 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:31.653338 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:31.673981 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:31.789656 306 consensus_queue.cc:237] T e165c71f8b6645a6936b58b2a8da1842 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:31.802037 30135 raft_consensus.cc:1261] T e165c71f8b6645a6936b58b2a8da1842 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:31.802047 30264 raft_consensus.cc:1261] T e165c71f8b6645a6936b58b2a8da1842 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:31.802564 5176 consensus_queue.cc:1035] T e165c71f8b6645a6936b58b2a8da1842 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:31.802894 4985 consensus_queue.cc:1035] T e165c71f8b6645a6936b58b2a8da1842 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:31.805397 4985 raft_consensus.cc:2932] T e165c71f8b6645a6936b58b2a8da1842 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:31.806233 30135 raft_consensus.cc:2932] T e165c71f8b6645a6936b58b2a8da1842 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:31.811838 29725 catalog_manager.cc:5484] T e165c71f8b6645a6936b58b2a8da1842 P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:31.813799 30264 raft_consensus.cc:2932] T e165c71f8b6645a6936b58b2a8da1842 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:31.837373 547 tablet_service.cc:1514] Processing DeleteTablet for tablet e165c71f8b6645a6936b58b2a8da1842 with delete_type TABLET_DATA_TOMBSTONED (TS f575ce92c0224eeb80b9fae5f6561fbe not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:39028
I20240503 02:13:31.838640 5414 tablet_replica.cc:330] T e165c71f8b6645a6936b58b2a8da1842 P f575ce92c0224eeb80b9fae5f6561fbe: stopping tablet replica
I20240503 02:13:31.842279 5414 raft_consensus.cc:2229] T e165c71f8b6645a6936b58b2a8da1842 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:31.842568 5414 raft_consensus.cc:2258] T e165c71f8b6645a6936b58b2a8da1842 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:31.843132 5414 ts_tablet_manager.cc:1875] T e165c71f8b6645a6936b58b2a8da1842 P f575ce92c0224eeb80b9fae5f6561fbe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:31.849491 5414 ts_tablet_manager.cc:1888] T e165c71f8b6645a6936b58b2a8da1842 P f575ce92c0224eeb80b9fae5f6561fbe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:13:31.849831 5414 log.cc:1196] T e165c71f8b6645a6936b58b2a8da1842 P f575ce92c0224eeb80b9fae5f6561fbe: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal/wals/e165c71f8b6645a6936b58b2a8da1842
I20240503 02:13:31.850694 29718 catalog_manager.cc:4830] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): tablet e165c71f8b6645a6936b58b2a8da1842 (table rebalancer_extra_table_34 [id=2274af2004634409884d3bcbca0009ca]) successfully deleted
I20240503 02:13:32.198438 29853 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:32.234966 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:32.235419 32755 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:32.239914 30244 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:32.281347 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:32.339170 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:32.388504 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:32.476990 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:32.576081 433 consensus_queue.cc:237] T 83e8dad806564b128d5f0fe559d8ffe1 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:32.577937 30005 raft_consensus.cc:1261] T 83e8dad806564b128d5f0fe559d8ffe1 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:32.577966 30132 raft_consensus.cc:1261] T 83e8dad806564b128d5f0fe559d8ffe1 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:32.578455 5373 consensus_queue.cc:1035] T 83e8dad806564b128d5f0fe559d8ffe1 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:32.579197 4830 consensus_queue.cc:1035] T 83e8dad806564b128d5f0fe559d8ffe1 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:32.581602 5373 raft_consensus.cc:2932] T 83e8dad806564b128d5f0fe559d8ffe1 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:32.584924 30132 raft_consensus.cc:2932] T 83e8dad806564b128d5f0fe559d8ffe1 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:32.588340 29727 catalog_manager.cc:5484] T 83e8dad806564b128d5f0fe559d8ffe1 P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:32.598630 30005 raft_consensus.cc:2932] T 83e8dad806564b128d5f0fe559d8ffe1 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:32.617740 32619 tablet_service.cc:1514] Processing DeleteTablet for tablet 83e8dad806564b128d5f0fe559d8ffe1 with delete_type TABLET_DATA_TOMBSTONED (TS 9a4d92bc25374fb3960fde7d10be309d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:38200
I20240503 02:13:32.618553 5507 tablet_replica.cc:330] T 83e8dad806564b128d5f0fe559d8ffe1 P 9a4d92bc25374fb3960fde7d10be309d: stopping tablet replica
I20240503 02:13:32.620138 5507 raft_consensus.cc:2229] T 83e8dad806564b128d5f0fe559d8ffe1 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:32.620445 5507 raft_consensus.cc:2258] T 83e8dad806564b128d5f0fe559d8ffe1 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:32.621169 5507 ts_tablet_manager.cc:1875] T 83e8dad806564b128d5f0fe559d8ffe1 P 9a4d92bc25374fb3960fde7d10be309d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:32.627302 5507 ts_tablet_manager.cc:1888] T 83e8dad806564b128d5f0fe559d8ffe1 P 9a4d92bc25374fb3960fde7d10be309d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:13:32.627437 5507 log.cc:1196] T 83e8dad806564b128d5f0fe559d8ffe1 P 9a4d92bc25374fb3960fde7d10be309d: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal/wals/83e8dad806564b128d5f0fe559d8ffe1
I20240503 02:13:32.628237 29719 catalog_manager.cc:4830] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): tablet 83e8dad806564b128d5f0fe559d8ffe1 (table rebalancer_extra_table_26 [id=9a18661b2f054802a39965b2fb3ce514]) successfully deleted
I20240503 02:13:32.877954 29853 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:32.880409 30244 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:32.918509 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:32.960659 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:33.012140 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:33.013252 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:33.016153 32755 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:33.056087 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240503 02:13:33.099653 32709 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.5:35595, user_credentials={real_user=slave}} blocked reactor thread for 50416.9us
W20240503 02:13:33.105374 32708 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.8:42233, user_credentials={real_user=slave}} blocked reactor thread for 55476.3us
I20240503 02:13:33.206846 30132 consensus_queue.cc:237] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:33.208747 29873 raft_consensus.cc:1261] T 5040203bd48549ff9c8f64c71d6bc598 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:33.209354 565 raft_consensus.cc:1261] T 5040203bd48549ff9c8f64c71d6bc598 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:33.209760 5473 consensus_queue.cc:1035] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:33.210058 5068 consensus_queue.cc:1035] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:33.212719 5068 raft_consensus.cc:2932] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:33.213671 29873 raft_consensus.cc:2932] T 5040203bd48549ff9c8f64c71d6bc598 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:33.218539 565 raft_consensus.cc:2932] T 5040203bd48549ff9c8f64c71d6bc598 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:33.219400 29730 catalog_manager.cc:5484] T 5040203bd48549ff9c8f64c71d6bc598 P 16440e365430405fb10366ab70e2b429 reported cstate change: config changed from index 4 to 5, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) evicted. New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:33.227002 30244 tablet_service.cc:1514] Processing DeleteTablet for tablet 5040203bd48549ff9c8f64c71d6bc598 with delete_type TABLET_DATA_TOMBSTONED (TS 1f10f80f13c84e80b03eda7085a3b012 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:49856
I20240503 02:13:33.229753 5590 tablet_replica.cc:330] T 5040203bd48549ff9c8f64c71d6bc598 P 1f10f80f13c84e80b03eda7085a3b012: stopping tablet replica
I20240503 02:13:33.229962 5590 raft_consensus.cc:2229] T 5040203bd48549ff9c8f64c71d6bc598 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:33.230217 5590 raft_consensus.cc:2258] T 5040203bd48549ff9c8f64c71d6bc598 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:33.230870 5590 ts_tablet_manager.cc:1875] T 5040203bd48549ff9c8f64c71d6bc598 P 1f10f80f13c84e80b03eda7085a3b012: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:33.237138 5590 ts_tablet_manager.cc:1888] T 5040203bd48549ff9c8f64c71d6bc598 P 1f10f80f13c84e80b03eda7085a3b012: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:13:33.237326 5590 log.cc:1196] T 5040203bd48549ff9c8f64c71d6bc598 P 1f10f80f13c84e80b03eda7085a3b012: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/wals/5040203bd48549ff9c8f64c71d6bc598
I20240503 02:13:33.238129 29719 catalog_manager.cc:4830] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): tablet 5040203bd48549ff9c8f64c71d6bc598 (table rebalance_test_table_19 [id=3c36e49b52c04307a655de366327db8c]) successfully deleted
I20240503 02:13:33.613421 29853 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:33.637228 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:33.648999 30244 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:33.673693 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:33.674124 32755 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:33.681234 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:33.711472 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:33.769043 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:33.887454 306 consensus_queue.cc:237] T 00aedc4902a745d6a859cbbf99529a7f P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:33.889451 30266 raft_consensus.cc:1261] T 00aedc4902a745d6a859cbbf99529a7f P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:33.889474 30005 raft_consensus.cc:1261] T 00aedc4902a745d6a859cbbf99529a7f P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:33.890139 5562 consensus_queue.cc:1035] T 00aedc4902a745d6a859cbbf99529a7f P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:33.890450 5176 consensus_queue.cc:1035] T 00aedc4902a745d6a859cbbf99529a7f P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:33.892966 5176 raft_consensus.cc:2932] T 00aedc4902a745d6a859cbbf99529a7f P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:33.893581 30266 raft_consensus.cc:2932] T 00aedc4902a745d6a859cbbf99529a7f P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:33.904250 29732 catalog_manager.cc:5484] T 00aedc4902a745d6a859cbbf99529a7f P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:33.929689 30005 raft_consensus.cc:2932] T 00aedc4902a745d6a859cbbf99529a7f P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:33.941547 32619 tablet_service.cc:1514] Processing DeleteTablet for tablet 00aedc4902a745d6a859cbbf99529a7f with delete_type TABLET_DATA_TOMBSTONED (TS 9a4d92bc25374fb3960fde7d10be309d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:38200
I20240503 02:13:33.945717 5682 tablet_replica.cc:330] T 00aedc4902a745d6a859cbbf99529a7f P 9a4d92bc25374fb3960fde7d10be309d: stopping tablet replica
I20240503 02:13:33.945958 5682 raft_consensus.cc:2229] T 00aedc4902a745d6a859cbbf99529a7f P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:33.947079 5682 raft_consensus.cc:2258] T 00aedc4902a745d6a859cbbf99529a7f P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:33.947821 5682 ts_tablet_manager.cc:1875] T 00aedc4902a745d6a859cbbf99529a7f P 9a4d92bc25374fb3960fde7d10be309d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:33.956435 5682 ts_tablet_manager.cc:1888] T 00aedc4902a745d6a859cbbf99529a7f P 9a4d92bc25374fb3960fde7d10be309d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:13:33.956600 5682 log.cc:1196] T 00aedc4902a745d6a859cbbf99529a7f P 9a4d92bc25374fb3960fde7d10be309d: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal/wals/00aedc4902a745d6a859cbbf99529a7f
I20240503 02:13:33.957392 29719 catalog_manager.cc:4830] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): tablet 00aedc4902a745d6a859cbbf99529a7f (table rebalancer_extra_table_2 [id=27940b10e50e4a418d9e7155f5a25394]) successfully deleted
I20240503 02:13:34.273823 29853 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:34.281064 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:34.301149 32755 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 02:13:34.341305 32573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.8:42233, user_credentials={real_user=slave}} blocked reactor thread for 57839.5us
I20240503 02:13:34.354996 30244 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:34.367015 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:34.371361 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:34.376775 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:34.422516 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:34.588407 434 consensus_queue.cc:237] T ec3891bfc4014b63afef6c259de00ca7 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:34.590392 29872 raft_consensus.cc:1261] T ec3891bfc4014b63afef6c259de00ca7 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:34.590461 305 raft_consensus.cc:1261] T ec3891bfc4014b63afef6c259de00ca7 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:34.590868 5718 consensus_queue.cc:1035] T ec3891bfc4014b63afef6c259de00ca7 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:34.591260 4830 consensus_queue.cc:1035] T ec3891bfc4014b63afef6c259de00ca7 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:34.594210 4830 raft_consensus.cc:2932] T ec3891bfc4014b63afef6c259de00ca7 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:34.598572 29873 raft_consensus.cc:2932] T ec3891bfc4014b63afef6c259de00ca7 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:34.600730 29729 catalog_manager.cc:5484] T ec3891bfc4014b63afef6c259de00ca7 P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:34.616016 32619 tablet_service.cc:1514] Processing DeleteTablet for tablet ec3891bfc4014b63afef6c259de00ca7 with delete_type TABLET_DATA_TOMBSTONED (TS 9a4d92bc25374fb3960fde7d10be309d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:38200
I20240503 02:13:34.616997 5772 tablet_replica.cc:330] T ec3891bfc4014b63afef6c259de00ca7 P 9a4d92bc25374fb3960fde7d10be309d: stopping tablet replica
I20240503 02:13:34.617326 5772 raft_consensus.cc:2229] T ec3891bfc4014b63afef6c259de00ca7 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:34.618253 5772 raft_consensus.cc:2258] T ec3891bfc4014b63afef6c259de00ca7 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:34.618736 306 raft_consensus.cc:2932] T ec3891bfc4014b63afef6c259de00ca7 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:34.619225 5772 ts_tablet_manager.cc:1875] T ec3891bfc4014b63afef6c259de00ca7 P 9a4d92bc25374fb3960fde7d10be309d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:34.625262 5772 ts_tablet_manager.cc:1888] T ec3891bfc4014b63afef6c259de00ca7 P 9a4d92bc25374fb3960fde7d10be309d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:13:34.625388 5772 log.cc:1196] T ec3891bfc4014b63afef6c259de00ca7 P 9a4d92bc25374fb3960fde7d10be309d: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal/wals/ec3891bfc4014b63afef6c259de00ca7
I20240503 02:13:34.626058 29719 catalog_manager.cc:4830] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): tablet ec3891bfc4014b63afef6c259de00ca7 (table rebalancer_extra_table_36 [id=9f19e29c56ab440eb34df78f91d5bd29]) successfully deleted
W20240503 02:13:34.862382 30199 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.3:42679, user_credentials={real_user=slave}} blocked reactor thread for 44636.5us
I20240503 02:13:34.911931 29853 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:34.960980 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:34.961112 30244 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:35.019474 32755 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:35.081488 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:35.091708 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:35.102639 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:35.128720 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:35.249817 29873 consensus_queue.cc:237] T 1758ec3783b24d3c820ed0d6dc118e69 P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:35.251855 30265 raft_consensus.cc:1261] T 1758ec3783b24d3c820ed0d6dc118e69 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:35.252385 5465 consensus_queue.cc:1035] T 1758ec3783b24d3c820ed0d6dc118e69 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:35.253401 30005 raft_consensus.cc:1261] T 1758ec3783b24d3c820ed0d6dc118e69 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:35.253957 5465 consensus_queue.cc:1035] T 1758ec3783b24d3c820ed0d6dc118e69 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:35.256101 5619 raft_consensus.cc:2932] T 1758ec3783b24d3c820ed0d6dc118e69 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:35.256814 30265 raft_consensus.cc:2932] T 1758ec3783b24d3c820ed0d6dc118e69 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:35.259042 30005 raft_consensus.cc:2932] T 1758ec3783b24d3c820ed0d6dc118e69 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:35.262197 29732 catalog_manager.cc:5484] T 1758ec3783b24d3c820ed0d6dc118e69 P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:35.272845 415 tablet_service.cc:1514] Processing DeleteTablet for tablet 1758ec3783b24d3c820ed0d6dc118e69 with delete_type TABLET_DATA_TOMBSTONED (TS fbbc2be6382e47908ecb7c68c367bd26 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50814
I20240503 02:13:35.274646 5870 tablet_replica.cc:330] T 1758ec3783b24d3c820ed0d6dc118e69 P fbbc2be6382e47908ecb7c68c367bd26: stopping tablet replica
I20240503 02:13:35.274920 5870 raft_consensus.cc:2229] T 1758ec3783b24d3c820ed0d6dc118e69 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:35.275141 5870 raft_consensus.cc:2258] T 1758ec3783b24d3c820ed0d6dc118e69 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:35.275768 5870 ts_tablet_manager.cc:1875] T 1758ec3783b24d3c820ed0d6dc118e69 P fbbc2be6382e47908ecb7c68c367bd26: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:35.282652 5870 ts_tablet_manager.cc:1888] T 1758ec3783b24d3c820ed0d6dc118e69 P fbbc2be6382e47908ecb7c68c367bd26: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:13:35.282788 5870 log.cc:1196] T 1758ec3783b24d3c820ed0d6dc118e69 P fbbc2be6382e47908ecb7c68c367bd26: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal/wals/1758ec3783b24d3c820ed0d6dc118e69
I20240503 02:13:35.283593 29719 catalog_manager.cc:4830] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): tablet 1758ec3783b24d3c820ed0d6dc118e69 (table rebalancer_extra_table_42 [id=b579d0acbdab49ba82f3db319ec150ce]) successfully deleted
I20240503 02:13:35.607537 29853 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:35.609124 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:35.620013 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:35.633982 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:35.693316 30244 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:35.715852 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:35.717628 32755 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:35.747869 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:35.838541 30266 consensus_queue.cc:237] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:35.840479 32640 raft_consensus.cc:1261] T 9fa3f1e452c24c7d83a36f10284b8b5d P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:35.840487 30135 raft_consensus.cc:1261] T 9fa3f1e452c24c7d83a36f10284b8b5d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:35.840999 5825 consensus_queue.cc:1035] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:35.841282 5564 consensus_queue.cc:1035] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:35.843644 5564 raft_consensus.cc:2932] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:35.844288 32640 raft_consensus.cc:2932] T 9fa3f1e452c24c7d83a36f10284b8b5d P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:35.851351 29730 catalog_manager.cc:5484] T 9fa3f1e452c24c7d83a36f10284b8b5d P 9a4d92bc25374fb3960fde7d10be309d reported cstate change: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:35.881879 30135 raft_consensus.cc:2932] T 9fa3f1e452c24c7d83a36f10284b8b5d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:35.904232 415 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fa3f1e452c24c7d83a36f10284b8b5d with delete_type TABLET_DATA_TOMBSTONED (TS fbbc2be6382e47908ecb7c68c367bd26 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50814
I20240503 02:13:35.924142 5957 tablet_replica.cc:330] T 9fa3f1e452c24c7d83a36f10284b8b5d P fbbc2be6382e47908ecb7c68c367bd26: stopping tablet replica
I20240503 02:13:35.924365 5957 raft_consensus.cc:2229] T 9fa3f1e452c24c7d83a36f10284b8b5d P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:35.924568 5957 raft_consensus.cc:2258] T 9fa3f1e452c24c7d83a36f10284b8b5d P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:35.925290 5957 ts_tablet_manager.cc:1875] T 9fa3f1e452c24c7d83a36f10284b8b5d P fbbc2be6382e47908ecb7c68c367bd26: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:35.936305 5957 ts_tablet_manager.cc:1888] T 9fa3f1e452c24c7d83a36f10284b8b5d P fbbc2be6382e47908ecb7c68c367bd26: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:13:35.936590 5957 log.cc:1196] T 9fa3f1e452c24c7d83a36f10284b8b5d P fbbc2be6382e47908ecb7c68c367bd26: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal/wals/9fa3f1e452c24c7d83a36f10284b8b5d
I20240503 02:13:35.937731 29719 catalog_manager.cc:4830] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): tablet 9fa3f1e452c24c7d83a36f10284b8b5d (table rebalancer_extra_table_16 [id=97e0c0e7261b408f9cb42ad58ba75222]) successfully deleted
I20240503 02:13:36.208588 29853 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:36.241261 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:36.242334 30244 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:36.243561 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:36.290161 32755 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:36.295629 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:36.315692 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:36.323700 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:36.542918 29873 consensus_queue.cc:237] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:36.544921 30135 raft_consensus.cc:1261] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:36.545039 30005 raft_consensus.cc:1261] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:36.545480 5465 consensus_queue.cc:1035] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:36.545786 5619 consensus_queue.cc:1035] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:36.548224 5619 raft_consensus.cc:2932] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:36.549584 30005 raft_consensus.cc:2932] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:36.550555 30135 raft_consensus.cc:2932] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:36.554865 29726 catalog_manager.cc:5484] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P 16440e365430405fb10366ab70e2b429 reported cstate change: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:36.569818 547 tablet_service.cc:1514] Processing DeleteTablet for tablet 70b8ebcf6f3245e6bd93eeb38ca1cbf5 with delete_type TABLET_DATA_TOMBSTONED (TS f575ce92c0224eeb80b9fae5f6561fbe not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:39028
I20240503 02:13:36.570629 6038 tablet_replica.cc:330] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P f575ce92c0224eeb80b9fae5f6561fbe: stopping tablet replica
I20240503 02:13:36.570923 6038 raft_consensus.cc:2229] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:36.571214 6038 raft_consensus.cc:2258] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:36.572142 6038 ts_tablet_manager.cc:1875] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P f575ce92c0224eeb80b9fae5f6561fbe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:36.577297 6038 ts_tablet_manager.cc:1888] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P f575ce92c0224eeb80b9fae5f6561fbe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:13:36.577397 6038 log.cc:1196] T 70b8ebcf6f3245e6bd93eeb38ca1cbf5 P f575ce92c0224eeb80b9fae5f6561fbe: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal/wals/70b8ebcf6f3245e6bd93eeb38ca1cbf5
I20240503 02:13:36.578127 29718 catalog_manager.cc:4830] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): tablet 70b8ebcf6f3245e6bd93eeb38ca1cbf5 (table rebalancer_extra_table_6 [id=53af41732fcb40aea2f1c6f0ac294bde]) successfully deleted
I20240503 02:13:36.828197 29853 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:36.860147 30244 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:36.860806 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:36.882580 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:36.932852 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:36.938793 32755 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:36.939181 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:36.970502 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 02:13:36.991742 29942 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.1:36719, user_credentials={real_user=slave}} blocked reactor thread for 59533.8us
I20240503 02:13:37.171728 32640 consensus_queue.cc:237] T 0ec95509de3443bd96bf91a08ef7df4a P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:37.178364 29873 raft_consensus.cc:1261] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:37.178490 565 raft_consensus.cc:1261] T 0ec95509de3443bd96bf91a08ef7df4a P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:37.179620 4915 consensus_queue.cc:1035] T 0ec95509de3443bd96bf91a08ef7df4a P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:37.180107 5898 consensus_queue.cc:1035] T 0ec95509de3443bd96bf91a08ef7df4a P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:37.184008 4915 raft_consensus.cc:2932] T 0ec95509de3443bd96bf91a08ef7df4a P 9a4d92bc25374fb3960fde7d10be309d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:37.184818 564 raft_consensus.cc:2932] T 0ec95509de3443bd96bf91a08ef7df4a P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:37.191588 29873 raft_consensus.cc:2932] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:37.198596 29733 catalog_manager.cc:5484] T 0ec95509de3443bd96bf91a08ef7df4a P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: config changed from index 2 to 3, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New cstate: current_term: 1 leader_uuid: "9a4d92bc25374fb3960fde7d10be309d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:37.211629 32755 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ec95509de3443bd96bf91a08ef7df4a with delete_type TABLET_DATA_TOMBSTONED (TS 8a666e798fe8477d8dd1e51b385d5ead not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46360
I20240503 02:13:37.212188 6130 tablet_replica.cc:330] T 0ec95509de3443bd96bf91a08ef7df4a P 8a666e798fe8477d8dd1e51b385d5ead: stopping tablet replica
I20240503 02:13:37.212373 6130 raft_consensus.cc:2229] T 0ec95509de3443bd96bf91a08ef7df4a P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:37.212563 6130 raft_consensus.cc:2258] T 0ec95509de3443bd96bf91a08ef7df4a P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:37.213421 6130 ts_tablet_manager.cc:1875] T 0ec95509de3443bd96bf91a08ef7df4a P 8a666e798fe8477d8dd1e51b385d5ead: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:37.220085 6130 ts_tablet_manager.cc:1888] T 0ec95509de3443bd96bf91a08ef7df4a P 8a666e798fe8477d8dd1e51b385d5ead: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:13:37.220324 6130 log.cc:1196] T 0ec95509de3443bd96bf91a08ef7df4a P 8a666e798fe8477d8dd1e51b385d5ead: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal/wals/0ec95509de3443bd96bf91a08ef7df4a
I20240503 02:13:37.221117 29719 catalog_manager.cc:4830] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): tablet 0ec95509de3443bd96bf91a08ef7df4a (table rebalancer_extra_table_22 [id=fedaa5e192b44061a6bd08563b273009]) successfully deleted
I20240503 02:13:37.539906 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:37.558458 29853 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:37.620215 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:37.659031 32755 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:37.662308 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:37.675611 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:37.715569 30244 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:37.715665 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:37.859210 30004 consensus_queue.cc:237] T 63b416f4e004490b8e38a96fd065dfea P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:37.861464 434 raft_consensus.cc:1261] T 63b416f4e004490b8e38a96fd065dfea P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:37.861456 29873 raft_consensus.cc:1261] T 63b416f4e004490b8e38a96fd065dfea P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:37.861971 6104 consensus_queue.cc:1035] T 63b416f4e004490b8e38a96fd065dfea P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:37.862244 5636 consensus_queue.cc:1035] T 63b416f4e004490b8e38a96fd065dfea P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:37.864794 6104 raft_consensus.cc:2932] T 63b416f4e004490b8e38a96fd065dfea P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:37.865375 29873 raft_consensus.cc:2932] T 63b416f4e004490b8e38a96fd065dfea P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:37.868526 434 raft_consensus.cc:2932] T 63b416f4e004490b8e38a96fd065dfea P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:37.872670 29732 catalog_manager.cc:5484] T 63b416f4e004490b8e38a96fd065dfea P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:37.922360 30114 tablet_service.cc:1514] Processing DeleteTablet for tablet 63b416f4e004490b8e38a96fd065dfea with delete_type TABLET_DATA_TOMBSTONED (TS a9094240860e43bfb129193f3bfd5be6 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44812
I20240503 02:13:37.930812 6224 tablet_replica.cc:330] T 63b416f4e004490b8e38a96fd065dfea P a9094240860e43bfb129193f3bfd5be6: stopping tablet replica
I20240503 02:13:37.931197 6224 raft_consensus.cc:2229] T 63b416f4e004490b8e38a96fd065dfea P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:37.931483 6224 raft_consensus.cc:2258] T 63b416f4e004490b8e38a96fd065dfea P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:37.932418 6224 ts_tablet_manager.cc:1875] T 63b416f4e004490b8e38a96fd065dfea P a9094240860e43bfb129193f3bfd5be6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:37.951831 6224 ts_tablet_manager.cc:1888] T 63b416f4e004490b8e38a96fd065dfea P a9094240860e43bfb129193f3bfd5be6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:13:37.952085 6224 log.cc:1196] T 63b416f4e004490b8e38a96fd065dfea P a9094240860e43bfb129193f3bfd5be6: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/wals/63b416f4e004490b8e38a96fd065dfea
I20240503 02:13:37.952857 29718 catalog_manager.cc:4830] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): tablet 63b416f4e004490b8e38a96fd065dfea (table rebalance_test_table_15 [id=6bbf46811ff140d289e600493149f118]) successfully deleted
I20240503 02:13:38.259271 29853 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:38.288906 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:38.311542 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:38.313220 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:38.341974 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:38.342746 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:38.410845 30244 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:38.412988 32755 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:38.580610 32638 consensus_queue.cc:237] T 58e1aaecf47944569296fc1e0780efb5 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:38.590667 30266 raft_consensus.cc:1261] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:13:38.590739 433 raft_consensus.cc:1261] T 58e1aaecf47944569296fc1e0780efb5 P fbbc2be6382e47908ecb7c68c367bd26 [term 2 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:13:38.591336 6125 consensus_queue.cc:1035] T 58e1aaecf47944569296fc1e0780efb5 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:38.592283 5898 consensus_queue.cc:1035] T 58e1aaecf47944569296fc1e0780efb5 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:38.596055 6311 raft_consensus.cc:2932] T 58e1aaecf47944569296fc1e0780efb5 P 9a4d92bc25374fb3960fde7d10be309d [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:38.597075 433 raft_consensus.cc:2932] T 58e1aaecf47944569296fc1e0780efb5 P fbbc2be6382e47908ecb7c68c367bd26 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:38.597632 30266 raft_consensus.cc:2932] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:38.603266 29727 catalog_manager.cc:5484] T 58e1aaecf47944569296fc1e0780efb5 P 9a4d92bc25374fb3960fde7d10be309d reported cstate change: config changed from index 2 to 4, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New cstate: current_term: 2 leader_uuid: "9a4d92bc25374fb3960fde7d10be309d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:38.612600 29853 tablet_service.cc:1514] Processing DeleteTablet for tablet 58e1aaecf47944569296fc1e0780efb5 with delete_type TABLET_DATA_TOMBSTONED (TS 16440e365430405fb10366ab70e2b429 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58744
I20240503 02:13:38.613212 6322 tablet_replica.cc:330] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:13:38.613510 6322 raft_consensus.cc:2229] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:38.613703 6322 raft_consensus.cc:2258] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:38.614693 6322 ts_tablet_manager.cc:1875] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:38.620523 6322 ts_tablet_manager.cc:1888] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240503 02:13:38.620635 6322 log.cc:1196] T 58e1aaecf47944569296fc1e0780efb5 P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/58e1aaecf47944569296fc1e0780efb5
I20240503 02:13:38.621315 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet 58e1aaecf47944569296fc1e0780efb5 (table rebalancer_extra_table_32 [id=567dfe3064ef40a99eede42015bd1a35]) successfully deleted
W20240503 02:13:38.789687 30070 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.5:35595, user_credentials={real_user=slave}} blocked reactor thread for 51493us
I20240503 02:13:38.857293 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:38.865065 29853 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:38.887763 30244 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:38.942458 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:39.017843 32755 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:39.024968 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:39.041204 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:39.050282 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:39.166055 29872 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "89938d4c793a44408795398078cc2ee0"
dest_uuid: "16440e365430405fb10366ab70e2b429"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56324
I20240503 02:13:39.166307 29872 raft_consensus.cc:595] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:13:39.167657 5465 raft_consensus.cc:979] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429: : Instructing follower 1f10f80f13c84e80b03eda7085a3b012 to start an election
I20240503 02:13:39.167768 5465 raft_consensus.cc:1067] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Signalling peer 1f10f80f13c84e80b03eda7085a3b012 to start an election
I20240503 02:13:39.168534 30265 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "89938d4c793a44408795398078cc2ee0"
dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
from {username='slave'} at 127.22.108.1:38817
I20240503 02:13:39.168731 30265 raft_consensus.cc:482] T 89938d4c793a44408795398078cc2ee0 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:39.168853 30265 raft_consensus.cc:3037] T 89938d4c793a44408795398078cc2ee0 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:39.171628 30265 raft_consensus.cc:504] T 89938d4c793a44408795398078cc2ee0 P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:39.172962 29872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89938d4c793a44408795398078cc2ee0" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:13:39.173171 29872 raft_consensus.cc:3032] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:39.173230 29872 raft_consensus.cc:729] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Running, Role: LEADER
I20240503 02:13:39.173535 29872 consensus_queue.cc:260] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:39.174012 29872 raft_consensus.cc:3037] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:39.174172 30265 leader_election.cc:288] T 89938d4c793a44408795398078cc2ee0 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 2 election: Requested vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:39.181921 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89938d4c793a44408795398078cc2ee0" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:13:39.182152 307 raft_consensus.cc:3037] T 89938d4c793a44408795398078cc2ee0 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:39.182343 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89938d4c793a44408795398078cc2ee0" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:13:39.182519 30004 raft_consensus.cc:3037] T 89938d4c793a44408795398078cc2ee0 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:39.184859 307 raft_consensus.cc:2454] T 89938d4c793a44408795398078cc2ee0 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 2.
I20240503 02:13:39.185142 30004 raft_consensus.cc:2454] T 89938d4c793a44408795398078cc2ee0 P 155735c7f8b14997ac4c770aee2a9401 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 2.
I20240503 02:13:39.186106 29872 raft_consensus.cc:2454] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 2.
I20240503 02:13:39.187018 30202 leader_election.cc:302] T 89938d4c793a44408795398078cc2ee0 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429, 1f10f80f13c84e80b03eda7085a3b012; no voters:
I20240503 02:13:39.187310 6372 raft_consensus.cc:2781] T 89938d4c793a44408795398078cc2ee0 P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:39.187424 6372 raft_consensus.cc:686] T 89938d4c793a44408795398078cc2ee0 P 1f10f80f13c84e80b03eda7085a3b012 [term 2 LEADER]: Becoming Leader. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Running, Role: LEADER
I20240503 02:13:39.187675 6372 consensus_queue.cc:237] T 89938d4c793a44408795398078cc2ee0 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:39.192124 29727 catalog_manager.cc:5484] T 89938d4c793a44408795398078cc2ee0 P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: term changed from 1 to 2, leader changed from 16440e365430405fb10366ab70e2b429 (127.22.108.1) to 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4). New cstate: current_term: 2 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:39.441282 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:39.459393 32755 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:39.484920 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:39.491000 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:39.550176 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:39.577739 30244 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:39.605203 305 raft_consensus.cc:1261] T 89938d4c793a44408795398078cc2ee0 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:13:39.605365 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:39.605759 6372 consensus_queue.cc:1035] T 89938d4c793a44408795398078cc2ee0 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:39.637816 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:39.773078 29873 raft_consensus.cc:1261] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429 [term 2 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:13:39.773619 6372 consensus_queue.cc:1035] T 89938d4c793a44408795398078cc2ee0 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:39.780730 30004 raft_consensus.cc:1261] T 89938d4c793a44408795398078cc2ee0 P 155735c7f8b14997ac4c770aee2a9401 [term 2 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:13:39.783998 6372 consensus_queue.cc:1035] T 89938d4c793a44408795398078cc2ee0 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:40.054081 30244 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:40.056998 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:40.058411 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:40.119506 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:40.147042 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:40.166989 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:40.171411 32755 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:40.239444 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:40.368734 29873 consensus_queue.cc:237] T 9102ab9545c54e82b642673795cb747a P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:40.385228 30135 raft_consensus.cc:1261] T 9102ab9545c54e82b642673795cb747a P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:40.386174 30004 raft_consensus.cc:1261] T 9102ab9545c54e82b642673795cb747a P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:40.387104 5619 consensus_queue.cc:1035] T 9102ab9545c54e82b642673795cb747a P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:40.387593 5619 consensus_queue.cc:1035] T 9102ab9545c54e82b642673795cb747a P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:40.447104 5619 raft_consensus.cc:2932] T 9102ab9545c54e82b642673795cb747a P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:40.451045 30004 raft_consensus.cc:2932] T 9102ab9545c54e82b642673795cb747a P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:40.457805 29734 catalog_manager.cc:5484] T 9102ab9545c54e82b642673795cb747a P 16440e365430405fb10366ab70e2b429 reported cstate change: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:40.452291 30135 raft_consensus.cc:2932] T 9102ab9545c54e82b642673795cb747a P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:40.493547 415 tablet_service.cc:1514] Processing DeleteTablet for tablet 9102ab9545c54e82b642673795cb747a with delete_type TABLET_DATA_TOMBSTONED (TS fbbc2be6382e47908ecb7c68c367bd26 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50814
I20240503 02:13:40.551587 6555 tablet_replica.cc:330] T 9102ab9545c54e82b642673795cb747a P fbbc2be6382e47908ecb7c68c367bd26: stopping tablet replica
I20240503 02:13:40.551903 6555 raft_consensus.cc:2229] T 9102ab9545c54e82b642673795cb747a P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:40.553129 6555 raft_consensus.cc:2258] T 9102ab9545c54e82b642673795cb747a P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:40.558959 6555 ts_tablet_manager.cc:1875] T 9102ab9545c54e82b642673795cb747a P fbbc2be6382e47908ecb7c68c367bd26: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:40.608088 6555 ts_tablet_manager.cc:1888] T 9102ab9545c54e82b642673795cb747a P fbbc2be6382e47908ecb7c68c367bd26: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:13:40.609474 6555 log.cc:1196] T 9102ab9545c54e82b642673795cb747a P fbbc2be6382e47908ecb7c68c367bd26: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal/wals/9102ab9545c54e82b642673795cb747a
I20240503 02:13:40.610277 29719 catalog_manager.cc:4830] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): tablet 9102ab9545c54e82b642673795cb747a (table rebalancer_extra_table_8 [id=c14bbf6b6bfd41148c82b54717aceb15]) successfully deleted
I20240503 02:13:40.870785 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:40.871270 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:40.899214 32755 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:40.909101 30244 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:40.958310 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:40.987694 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:41.020686 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:41.029500 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:41.175714 30005 consensus_queue.cc:237] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:41.177922 30265 raft_consensus.cc:1261] T 04e47da8cf7445dcacaf1f7c8950bd34 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:41.177947 435 raft_consensus.cc:1261] T 04e47da8cf7445dcacaf1f7c8950bd34 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:41.179054 6547 consensus_queue.cc:1035] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:41.179548 6547 consensus_queue.cc:1035] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:41.182534 6547 raft_consensus.cc:2932] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:41.186105 435 raft_consensus.cc:2932] T 04e47da8cf7445dcacaf1f7c8950bd34 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:41.190752 30266 raft_consensus.cc:2932] T 04e47da8cf7445dcacaf1f7c8950bd34 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:41.193332 29727 catalog_manager.cc:5484] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:41.208936 30114 tablet_service.cc:1514] Processing DeleteTablet for tablet 04e47da8cf7445dcacaf1f7c8950bd34 with delete_type TABLET_DATA_TOMBSTONED (TS a9094240860e43bfb129193f3bfd5be6 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44812
I20240503 02:13:41.209923 6649 tablet_replica.cc:330] T 04e47da8cf7445dcacaf1f7c8950bd34 P a9094240860e43bfb129193f3bfd5be6: stopping tablet replica
I20240503 02:13:41.210172 6649 raft_consensus.cc:2229] T 04e47da8cf7445dcacaf1f7c8950bd34 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:41.210491 6649 raft_consensus.cc:2258] T 04e47da8cf7445dcacaf1f7c8950bd34 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:41.211221 6649 ts_tablet_manager.cc:1875] T 04e47da8cf7445dcacaf1f7c8950bd34 P a9094240860e43bfb129193f3bfd5be6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:41.216360 6649 ts_tablet_manager.cc:1888] T 04e47da8cf7445dcacaf1f7c8950bd34 P a9094240860e43bfb129193f3bfd5be6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:13:41.216463 6649 log.cc:1196] T 04e47da8cf7445dcacaf1f7c8950bd34 P a9094240860e43bfb129193f3bfd5be6: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/wals/04e47da8cf7445dcacaf1f7c8950bd34
I20240503 02:13:41.217258 29718 catalog_manager.cc:4830] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): tablet 04e47da8cf7445dcacaf1f7c8950bd34 (table rebalance_test_table_11 [id=9498d0be60bf4564a9127c17a357f7ed]) successfully deleted
I20240503 02:13:41.517288 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:41.582855 32755 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:41.597211 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:41.641881 30244 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:41.666600 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:41.669657 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:41.673353 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:41.710616 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:41.813903 307 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f694fefb42d84318a6d91532105b3c90"
dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53224
I20240503 02:13:41.814137 307 raft_consensus.cc:595] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Received request to transfer leadership
I20240503 02:13:41.829156 6590 raft_consensus.cc:979] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead: : Instructing follower 1f10f80f13c84e80b03eda7085a3b012 to start an election
I20240503 02:13:41.829310 6590 raft_consensus.cc:1067] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Signalling peer 1f10f80f13c84e80b03eda7085a3b012 to start an election
I20240503 02:13:41.831880 30266 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f694fefb42d84318a6d91532105b3c90"
dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
from {username='slave'} at 127.22.108.6:54483
I20240503 02:13:41.832080 30266 raft_consensus.cc:482] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:41.832165 30266 raft_consensus.cc:3037] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:41.834906 30266 raft_consensus.cc:504] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:41.836081 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f694fefb42d84318a6d91532105b3c90" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:13:41.836148 29873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f694fefb42d84318a6d91532105b3c90" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:13:41.836272 307 raft_consensus.cc:3032] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:41.836334 307 raft_consensus.cc:729] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Running, Role: LEADER
I20240503 02:13:41.836359 29873 raft_consensus.cc:3037] T f694fefb42d84318a6d91532105b3c90 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:41.836632 307 consensus_queue.cc:260] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:41.837118 307 raft_consensus.cc:3037] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:41.837133 566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f694fefb42d84318a6d91532105b3c90" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:13:41.837318 566 raft_consensus.cc:3037] T f694fefb42d84318a6d91532105b3c90 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:41.839195 29873 raft_consensus.cc:2454] T f694fefb42d84318a6d91532105b3c90 P 16440e365430405fb10366ab70e2b429 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 2.
I20240503 02:13:41.839789 307 raft_consensus.cc:2454] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 2.
I20240503 02:13:41.839972 566 raft_consensus.cc:2454] T f694fefb42d84318a6d91532105b3c90 P f575ce92c0224eeb80b9fae5f6561fbe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 2.
I20240503 02:13:41.840253 30200 leader_election.cc:302] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, 1f10f80f13c84e80b03eda7085a3b012, 8a666e798fe8477d8dd1e51b385d5ead; no voters:
I20240503 02:13:41.840462 30266 leader_election.cc:288] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 2 election: Requested vote from peers 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:13:41.840521 6693 raft_consensus.cc:2781] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:41.840611 6693 raft_consensus.cc:686] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 [term 2 LEADER]: Becoming Leader. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Running, Role: LEADER
I20240503 02:13:41.840857 6693 consensus_queue.cc:237] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:41.845906 29732 catalog_manager.cc:5484] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: term changed from 1 to 2, leader changed from 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) to 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4). New cstate: current_term: 2 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:42.135054 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:42.138815 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:42.162339 30244 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:42.178341 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:42.181962 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:42.190833 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:42.247182 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:42.247356 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:42.331285 29873 raft_consensus.cc:1261] T f694fefb42d84318a6d91532105b3c90 P 16440e365430405fb10366ab70e2b429 [term 2 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:13:42.331919 6693 consensus_queue.cc:1035] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:42.413064 307 raft_consensus.cc:1261] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:13:42.413671 6475 consensus_queue.cc:1035] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:42.415386 305 consensus_queue.cc:237] T c87e7cbed0e843f5b5bb9bddef598cdb P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:42.418538 566 raft_consensus.cc:1261] T f694fefb42d84318a6d91532105b3c90 P f575ce92c0224eeb80b9fae5f6561fbe [term 2 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:13:42.419160 6693 consensus_queue.cc:1035] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:42.421226 30135 raft_consensus.cc:1261] T c87e7cbed0e843f5b5bb9bddef598cdb P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:13:42.421690 32709 consensus_peers.cc:485] T c87e7cbed0e843f5b5bb9bddef598cdb P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Couldn't send request to peer 155735c7f8b14997ac4c770aee2a9401. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c87e7cbed0e843f5b5bb9bddef598cdb. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:42.421859 6688 consensus_queue.cc:1035] T c87e7cbed0e843f5b5bb9bddef598cdb P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:42.447896 564 raft_consensus.cc:1261] T c87e7cbed0e843f5b5bb9bddef598cdb P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:42.448437 6797 consensus_queue.cc:1035] T c87e7cbed0e843f5b5bb9bddef598cdb P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:42.455823 6797 raft_consensus.cc:2932] T c87e7cbed0e843f5b5bb9bddef598cdb P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:42.463371 566 raft_consensus.cc:2932] T c87e7cbed0e843f5b5bb9bddef598cdb P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:42.469913 30135 raft_consensus.cc:2932] T c87e7cbed0e843f5b5bb9bddef598cdb P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:42.477607 29732 catalog_manager.cc:5484] T c87e7cbed0e843f5b5bb9bddef598cdb P a9094240860e43bfb129193f3bfd5be6 reported cstate change: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:42.538797 564 consensus_queue.cc:237] T 5800022ae50043ee8b1af1e134e86e6a P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
W20240503 02:13:42.541738 502 consensus_peers.cc:485] T 5800022ae50043ee8b1af1e134e86e6a P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): Couldn't send request to peer 1f10f80f13c84e80b03eda7085a3b012. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5800022ae50043ee8b1af1e134e86e6a. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:42.541929 32640 raft_consensus.cc:1261] T 5800022ae50043ee8b1af1e134e86e6a P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:42.541994 305 raft_consensus.cc:1261] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:42.542783 6821 consensus_queue.cc:1035] T 5800022ae50043ee8b1af1e134e86e6a P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:42.543159 6708 consensus_queue.cc:1035] T 5800022ae50043ee8b1af1e134e86e6a P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:42.547477 6821 raft_consensus.cc:2932] T 5800022ae50043ee8b1af1e134e86e6a P f575ce92c0224eeb80b9fae5f6561fbe [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:42.548039 305 raft_consensus.cc:2932] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:42.556154 29732 catalog_manager.cc:5484] T 5800022ae50043ee8b1af1e134e86e6a P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New cstate: current_term: 1 leader_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:42.556483 32640 raft_consensus.cc:2932] T 5800022ae50043ee8b1af1e134e86e6a P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:42.600998 30004 consensus_queue.cc:237] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:42.603437 30135 raft_consensus.cc:1261] T db0756f0f331402aaef0583e1f9e5318 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:42.603564 30265 raft_consensus.cc:1261] T db0756f0f331402aaef0583e1f9e5318 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:13:42.603749 29940 consensus_peers.cc:485] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): Couldn't send request to peer 8a666e798fe8477d8dd1e51b385d5ead. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db0756f0f331402aaef0583e1f9e5318. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:42.604241 6547 consensus_queue.cc:1035] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:42.604876 6638 consensus_queue.cc:1035] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:42.608057 6547 raft_consensus.cc:2932] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:42.611639 30265 raft_consensus.cc:2932] T db0756f0f331402aaef0583e1f9e5318 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:42.614285 30135 raft_consensus.cc:2932] T db0756f0f331402aaef0583e1f9e5318 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:42.615690 29726 catalog_manager.cc:5484] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: config changed from index -1 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:42.824512 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:42.842756 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:42.876009 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:42.907372 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:42.937145 30244 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:42.990199 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:43.024281 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:43.038129 6904 ts_tablet_manager.cc:902] T db0756f0f331402aaef0583e1f9e5318 P 8a666e798fe8477d8dd1e51b385d5ead: Initiating tablet copy from peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:13:43.062877 6906 ts_tablet_manager.cc:902] T 5800022ae50043ee8b1af1e134e86e6a P 1f10f80f13c84e80b03eda7085a3b012: Initiating tablet copy from peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:13:43.092530 6906 tablet_copy_client.cc:308] T 5800022ae50043ee8b1af1e134e86e6a P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.8:42233
I20240503 02:13:43.093235 588 tablet_copy_service.cc:140] P f575ce92c0224eeb80b9fae5f6561fbe: Received BeginTabletCopySession request for tablet 5800022ae50043ee8b1af1e134e86e6a from peer 1f10f80f13c84e80b03eda7085a3b012 ({username='slave'} at 127.22.108.4:45935)
I20240503 02:13:43.093401 588 tablet_copy_service.cc:161] P f575ce92c0224eeb80b9fae5f6561fbe: Beginning new tablet copy session on tablet 5800022ae50043ee8b1af1e134e86e6a from peer 1f10f80f13c84e80b03eda7085a3b012 at {username='slave'} at 127.22.108.4:45935: session id = 1f10f80f13c84e80b03eda7085a3b012-5800022ae50043ee8b1af1e134e86e6a
I20240503 02:13:43.097008 588 tablet_copy_source_session.cc:205] T 5800022ae50043ee8b1af1e134e86e6a P f575ce92c0224eeb80b9fae5f6561fbe: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:43.106112 6906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5800022ae50043ee8b1af1e134e86e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:43.120711 6906 tablet_copy_client.cc:783] T 5800022ae50043ee8b1af1e134e86e6a P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:43.121124 6906 tablet_copy_client.cc:647] T 5800022ae50043ee8b1af1e134e86e6a P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:43.124644 6906 tablet_copy_client.cc:523] T 5800022ae50043ee8b1af1e134e86e6a P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:43.139889 6902 ts_tablet_manager.cc:902] T c87e7cbed0e843f5b5bb9bddef598cdb P 155735c7f8b14997ac4c770aee2a9401: Initiating tablet copy from peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:43.140436 6906 tablet_bootstrap.cc:492] T 5800022ae50043ee8b1af1e134e86e6a P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:13:43.146764 6904 tablet_copy_client.cc:308] T db0756f0f331402aaef0583e1f9e5318 P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.2:37873
I20240503 02:13:43.147388 30026 tablet_copy_service.cc:140] P 155735c7f8b14997ac4c770aee2a9401: Received BeginTabletCopySession request for tablet db0756f0f331402aaef0583e1f9e5318 from peer 8a666e798fe8477d8dd1e51b385d5ead ({username='slave'} at 127.22.108.6:33785)
I20240503 02:13:43.147527 30026 tablet_copy_service.cc:161] P 155735c7f8b14997ac4c770aee2a9401: Beginning new tablet copy session on tablet db0756f0f331402aaef0583e1f9e5318 from peer 8a666e798fe8477d8dd1e51b385d5ead at {username='slave'} at 127.22.108.6:33785: session id = 8a666e798fe8477d8dd1e51b385d5ead-db0756f0f331402aaef0583e1f9e5318
I20240503 02:13:43.148367 6906 tablet_bootstrap.cc:492] T 5800022ae50043ee8b1af1e134e86e6a P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:43.148710 6906 tablet_bootstrap.cc:492] T 5800022ae50043ee8b1af1e134e86e6a P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap complete.
I20240503 02:13:43.148936 6906 ts_tablet_manager.cc:1367] T 5800022ae50043ee8b1af1e134e86e6a P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240503 02:13:43.149655 6906 raft_consensus.cc:348] T 5800022ae50043ee8b1af1e134e86e6a P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:43.149871 6906 raft_consensus.cc:729] T 5800022ae50043ee8b1af1e134e86e6a P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:13:43.150127 6906 consensus_queue.cc:260] T 5800022ae50043ee8b1af1e134e86e6a P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:43.150368 30026 tablet_copy_source_session.cc:205] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:43.150786 6906 ts_tablet_manager.cc:1398] T 5800022ae50043ee8b1af1e134e86e6a P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:13:43.151511 588 tablet_copy_service.cc:342] P f575ce92c0224eeb80b9fae5f6561fbe: Request end of tablet copy session 1f10f80f13c84e80b03eda7085a3b012-5800022ae50043ee8b1af1e134e86e6a received from {username='slave'} at 127.22.108.4:45935
I20240503 02:13:43.151641 588 tablet_copy_service.cc:433] P f575ce92c0224eeb80b9fae5f6561fbe: ending tablet copy session 1f10f80f13c84e80b03eda7085a3b012-5800022ae50043ee8b1af1e134e86e6a on tablet 5800022ae50043ee8b1af1e134e86e6a with peer 1f10f80f13c84e80b03eda7085a3b012
I20240503 02:13:43.152128 6904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db0756f0f331402aaef0583e1f9e5318. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:43.160544 6904 tablet_copy_client.cc:783] T db0756f0f331402aaef0583e1f9e5318 P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:43.160879 6904 tablet_copy_client.cc:647] T db0756f0f331402aaef0583e1f9e5318 P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:43.161370 6902 tablet_copy_client.cc:308] T c87e7cbed0e843f5b5bb9bddef598cdb P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.6:39975
I20240503 02:13:43.162268 327 tablet_copy_service.cc:140] P 8a666e798fe8477d8dd1e51b385d5ead: Received BeginTabletCopySession request for tablet c87e7cbed0e843f5b5bb9bddef598cdb from peer 155735c7f8b14997ac4c770aee2a9401 ({username='slave'} at 127.22.108.2:53863)
I20240503 02:13:43.162396 327 tablet_copy_service.cc:161] P 8a666e798fe8477d8dd1e51b385d5ead: Beginning new tablet copy session on tablet c87e7cbed0e843f5b5bb9bddef598cdb from peer 155735c7f8b14997ac4c770aee2a9401 at {username='slave'} at 127.22.108.2:53863: session id = 155735c7f8b14997ac4c770aee2a9401-c87e7cbed0e843f5b5bb9bddef598cdb
I20240503 02:13:43.163892 6904 tablet_copy_client.cc:523] T db0756f0f331402aaef0583e1f9e5318 P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:43.165246 327 tablet_copy_source_session.cc:205] T c87e7cbed0e843f5b5bb9bddef598cdb P 8a666e798fe8477d8dd1e51b385d5ead: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:43.166641 6902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c87e7cbed0e843f5b5bb9bddef598cdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:43.168924 6904 tablet_bootstrap.cc:492] T db0756f0f331402aaef0583e1f9e5318 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:43.169955 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:43.178812 6902 tablet_copy_client.cc:783] T c87e7cbed0e843f5b5bb9bddef598cdb P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:43.179749 6902 tablet_copy_client.cc:647] T c87e7cbed0e843f5b5bb9bddef598cdb P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:43.182206 6902 tablet_copy_client.cc:523] T c87e7cbed0e843f5b5bb9bddef598cdb P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:43.192443 6902 tablet_bootstrap.cc:492] T c87e7cbed0e843f5b5bb9bddef598cdb P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:13:43.220337 6902 tablet_bootstrap.cc:492] T c87e7cbed0e843f5b5bb9bddef598cdb P 155735c7f8b14997ac4c770aee2a9401: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:43.221112 6902 tablet_bootstrap.cc:492] T c87e7cbed0e843f5b5bb9bddef598cdb P 155735c7f8b14997ac4c770aee2a9401: Bootstrap complete.
I20240503 02:13:43.221464 6902 ts_tablet_manager.cc:1367] T c87e7cbed0e843f5b5bb9bddef598cdb P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.001s
I20240503 02:13:43.222167 6902 raft_consensus.cc:348] T c87e7cbed0e843f5b5bb9bddef598cdb P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:43.222433 6902 raft_consensus.cc:729] T c87e7cbed0e843f5b5bb9bddef598cdb P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:13:43.222688 6902 consensus_queue.cc:260] T c87e7cbed0e843f5b5bb9bddef598cdb P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:43.223332 6902 ts_tablet_manager.cc:1398] T c87e7cbed0e843f5b5bb9bddef598cdb P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:13:43.225555 327 tablet_copy_service.cc:342] P 8a666e798fe8477d8dd1e51b385d5ead: Request end of tablet copy session 155735c7f8b14997ac4c770aee2a9401-c87e7cbed0e843f5b5bb9bddef598cdb received from {username='slave'} at 127.22.108.2:53863
I20240503 02:13:43.225674 327 tablet_copy_service.cc:433] P 8a666e798fe8477d8dd1e51b385d5ead: ending tablet copy session 155735c7f8b14997ac4c770aee2a9401-c87e7cbed0e843f5b5bb9bddef598cdb on tablet c87e7cbed0e843f5b5bb9bddef598cdb with peer 155735c7f8b14997ac4c770aee2a9401
I20240503 02:13:43.266331 6904 tablet_bootstrap.cc:492] T db0756f0f331402aaef0583e1f9e5318 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:43.266920 6904 tablet_bootstrap.cc:492] T db0756f0f331402aaef0583e1f9e5318 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap complete.
I20240503 02:13:43.267231 6904 ts_tablet_manager.cc:1367] T db0756f0f331402aaef0583e1f9e5318 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.100s user 0.067s sys 0.003s
I20240503 02:13:43.268080 6904 raft_consensus.cc:348] T db0756f0f331402aaef0583e1f9e5318 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:43.268394 6904 raft_consensus.cc:729] T db0756f0f331402aaef0583e1f9e5318 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:43.268682 6904 consensus_queue.cc:260] T db0756f0f331402aaef0583e1f9e5318 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:43.269330 6904 ts_tablet_manager.cc:1398] T db0756f0f331402aaef0583e1f9e5318 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:43.270097 30026 tablet_copy_service.cc:342] P 155735c7f8b14997ac4c770aee2a9401: Request end of tablet copy session 8a666e798fe8477d8dd1e51b385d5ead-db0756f0f331402aaef0583e1f9e5318 received from {username='slave'} at 127.22.108.6:33785
I20240503 02:13:43.270238 30026 tablet_copy_service.cc:433] P 155735c7f8b14997ac4c770aee2a9401: ending tablet copy session 8a666e798fe8477d8dd1e51b385d5ead-db0756f0f331402aaef0583e1f9e5318 on tablet db0756f0f331402aaef0583e1f9e5318 with peer 8a666e798fe8477d8dd1e51b385d5ead
I20240503 02:13:43.516148 307 raft_consensus.cc:1203] T db0756f0f331402aaef0583e1f9e5318 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:43.553936 30265 raft_consensus.cc:1203] T 5800022ae50043ee8b1af1e134e86e6a P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:43.558252 30005 raft_consensus.cc:1203] T c87e7cbed0e843f5b5bb9bddef598cdb P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:43.671908 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:43.699061 30244 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:43.701373 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:43.713254 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:43.732123 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:43.762274 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:43.765208 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:43.798955 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:44.181361 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:44.199180 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:44.214622 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:44.233722 30244 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:44.256719 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:44.260555 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:44.325400 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:44.333250 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:44.506568 566 consensus_queue.cc:237] T 5800022ae50043ee8b1af1e134e86e6a P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:44.512399 30266 raft_consensus.cc:1261] T 5800022ae50043ee8b1af1e134e86e6a P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:44.512602 305 raft_consensus.cc:1261] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:44.513092 7012 consensus_queue.cc:1035] T 5800022ae50043ee8b1af1e134e86e6a P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:44.513432 7072 consensus_queue.cc:1035] T 5800022ae50043ee8b1af1e134e86e6a P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:44.517015 7012 raft_consensus.cc:2932] T 5800022ae50043ee8b1af1e134e86e6a P f575ce92c0224eeb80b9fae5f6561fbe [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:44.517607 305 raft_consensus.cc:2932] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:44.521296 30266 raft_consensus.cc:2932] T 5800022ae50043ee8b1af1e134e86e6a P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:44.526481 29727 catalog_manager.cc:5484] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New cstate: current_term: 1 leader_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:44.543624 32619 tablet_service.cc:1514] Processing DeleteTablet for tablet 5800022ae50043ee8b1af1e134e86e6a with delete_type TABLET_DATA_TOMBSTONED (TS 9a4d92bc25374fb3960fde7d10be309d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:38200
I20240503 02:13:44.544350 7085 tablet_replica.cc:330] T 5800022ae50043ee8b1af1e134e86e6a P 9a4d92bc25374fb3960fde7d10be309d: stopping tablet replica
I20240503 02:13:44.544679 7085 raft_consensus.cc:2229] T 5800022ae50043ee8b1af1e134e86e6a P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:44.545229 7085 raft_consensus.cc:2258] T 5800022ae50043ee8b1af1e134e86e6a P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:44.546021 7085 ts_tablet_manager.cc:1875] T 5800022ae50043ee8b1af1e134e86e6a P 9a4d92bc25374fb3960fde7d10be309d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:44.552433 7085 ts_tablet_manager.cc:1888] T 5800022ae50043ee8b1af1e134e86e6a P 9a4d92bc25374fb3960fde7d10be309d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:13:44.552528 7085 log.cc:1196] T 5800022ae50043ee8b1af1e134e86e6a P 9a4d92bc25374fb3960fde7d10be309d: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal/wals/5800022ae50043ee8b1af1e134e86e6a
I20240503 02:13:44.553210 29719 catalog_manager.cc:4830] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): tablet 5800022ae50043ee8b1af1e134e86e6a (table rebalancer_extra_table_26 [id=9a18661b2f054802a39965b2fb3ce514]) successfully deleted
W20240503 02:13:44.731930 32708 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.3:42679, user_credentials={real_user=slave}} blocked reactor thread for 58476.4us
I20240503 02:13:44.838891 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:44.850958 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:44.853469 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:44.869731 30244 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:44.871148 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:44.915598 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240503 02:13:44.942245 30202 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.1:36719, user_credentials={real_user=slave}} blocked reactor thread for 83288.5us
I20240503 02:13:44.950827 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:44.994375 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:45.155010 307 consensus_queue.cc:237] T c87e7cbed0e843f5b5bb9bddef598cdb P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:45.157013 30135 raft_consensus.cc:1261] T c87e7cbed0e843f5b5bb9bddef598cdb P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:45.158684 6689 consensus_queue.cc:1035] T c87e7cbed0e843f5b5bb9bddef598cdb P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:45.160154 30005 raft_consensus.cc:1261] T c87e7cbed0e843f5b5bb9bddef598cdb P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:45.160836 6689 consensus_queue.cc:1035] T c87e7cbed0e843f5b5bb9bddef598cdb P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:45.161481 7123 raft_consensus.cc:2932] T c87e7cbed0e843f5b5bb9bddef598cdb P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:45.162014 30133 raft_consensus.cc:2932] T c87e7cbed0e843f5b5bb9bddef598cdb P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:45.164389 30004 raft_consensus.cc:2932] T c87e7cbed0e843f5b5bb9bddef598cdb P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:45.168502 29725 catalog_manager.cc:5484] T c87e7cbed0e843f5b5bb9bddef598cdb P a9094240860e43bfb129193f3bfd5be6 reported cstate change: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:45.179586 547 tablet_service.cc:1514] Processing DeleteTablet for tablet c87e7cbed0e843f5b5bb9bddef598cdb with delete_type TABLET_DATA_TOMBSTONED (TS f575ce92c0224eeb80b9fae5f6561fbe not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:39028
I20240503 02:13:45.180200 7181 tablet_replica.cc:330] T c87e7cbed0e843f5b5bb9bddef598cdb P f575ce92c0224eeb80b9fae5f6561fbe: stopping tablet replica
I20240503 02:13:45.180411 7181 raft_consensus.cc:2229] T c87e7cbed0e843f5b5bb9bddef598cdb P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:45.180605 7181 raft_consensus.cc:2258] T c87e7cbed0e843f5b5bb9bddef598cdb P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:45.181322 7181 ts_tablet_manager.cc:1875] T c87e7cbed0e843f5b5bb9bddef598cdb P f575ce92c0224eeb80b9fae5f6561fbe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:45.187685 7181 ts_tablet_manager.cc:1888] T c87e7cbed0e843f5b5bb9bddef598cdb P f575ce92c0224eeb80b9fae5f6561fbe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:13:45.187795 7181 log.cc:1196] T c87e7cbed0e843f5b5bb9bddef598cdb P f575ce92c0224eeb80b9fae5f6561fbe: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal/wals/c87e7cbed0e843f5b5bb9bddef598cdb
I20240503 02:13:45.188553 29718 catalog_manager.cc:4830] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): tablet c87e7cbed0e843f5b5bb9bddef598cdb (table rebalancer_extra_table_38 [id=e4b9d13872ad4d27841df71e38d5482f]) successfully deleted
I20240503 02:13:45.492350 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:45.501111 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:45.503412 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:45.513540 30244 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:45.533854 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:45.534536 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:45.556546 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:45.573305 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:45.732120 32641 consensus_queue.cc:237] T 16cf978ac35341d4adea71a17a91194c P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:45.734493 29872 raft_consensus.cc:1261] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:45.734592 434 raft_consensus.cc:1261] T 16cf978ac35341d4adea71a17a91194c P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:45.735216 6900 consensus_queue.cc:1035] T 16cf978ac35341d4adea71a17a91194c P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:45.735471 7157 consensus_queue.cc:1035] T 16cf978ac35341d4adea71a17a91194c P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:45.738307 7157 raft_consensus.cc:2932] T 16cf978ac35341d4adea71a17a91194c P 9a4d92bc25374fb3960fde7d10be309d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:45.741880 29873 raft_consensus.cc:2932] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:45.746310 29730 catalog_manager.cc:5484] T 16cf978ac35341d4adea71a17a91194c P 9a4d92bc25374fb3960fde7d10be309d reported cstate change: config changed from index 2 to 3, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New cstate: current_term: 1 leader_uuid: "9a4d92bc25374fb3960fde7d10be309d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:45.762315 434 raft_consensus.cc:2932] T 16cf978ac35341d4adea71a17a91194c P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:45.770937 32755 tablet_service.cc:1514] Processing DeleteTablet for tablet 16cf978ac35341d4adea71a17a91194c with delete_type TABLET_DATA_TOMBSTONED (TS 8a666e798fe8477d8dd1e51b385d5ead not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46360
I20240503 02:13:45.771545 7276 tablet_replica.cc:330] T 16cf978ac35341d4adea71a17a91194c P 8a666e798fe8477d8dd1e51b385d5ead: stopping tablet replica
I20240503 02:13:45.771809 7276 raft_consensus.cc:2229] T 16cf978ac35341d4adea71a17a91194c P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:45.772068 7276 raft_consensus.cc:2258] T 16cf978ac35341d4adea71a17a91194c P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:45.772706 7276 ts_tablet_manager.cc:1875] T 16cf978ac35341d4adea71a17a91194c P 8a666e798fe8477d8dd1e51b385d5ead: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:45.781543 7276 ts_tablet_manager.cc:1888] T 16cf978ac35341d4adea71a17a91194c P 8a666e798fe8477d8dd1e51b385d5ead: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:13:45.781658 7276 log.cc:1196] T 16cf978ac35341d4adea71a17a91194c P 8a666e798fe8477d8dd1e51b385d5ead: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal/wals/16cf978ac35341d4adea71a17a91194c
I20240503 02:13:45.782343 29719 catalog_manager.cc:4830] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): tablet 16cf978ac35341d4adea71a17a91194c (table rebalancer_extra_table_26 [id=9a18661b2f054802a39965b2fb3ce514]) successfully deleted
I20240503 02:13:46.218036 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:46.239545 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:46.264657 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:46.266306 30244 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:46.295778 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:46.331511 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:46.336272 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:46.467320 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:46.590504 30265 consensus_queue.cc:237] T 89938d4c793a44408795398078cc2ee0 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:46.594446 307 raft_consensus.cc:1261] T 89938d4c793a44408795398078cc2ee0 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:13:46.594465 30004 raft_consensus.cc:1261] T 89938d4c793a44408795398078cc2ee0 P 155735c7f8b14997ac4c770aee2a9401 [term 2 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:13:46.594955 7394 consensus_queue.cc:1035] T 89938d4c793a44408795398078cc2ee0 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:46.595228 7154 consensus_queue.cc:1035] T 89938d4c793a44408795398078cc2ee0 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:46.598083 7241 raft_consensus.cc:2932] T 89938d4c793a44408795398078cc2ee0 P 1f10f80f13c84e80b03eda7085a3b012 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:46.598572 30004 raft_consensus.cc:2932] T 89938d4c793a44408795398078cc2ee0 P 155735c7f8b14997ac4c770aee2a9401 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:46.598769 305 raft_consensus.cc:2932] T 89938d4c793a44408795398078cc2ee0 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:46.603754 29731 catalog_manager.cc:5484] T 89938d4c793a44408795398078cc2ee0 P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: config changed from index 2 to 4, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New cstate: current_term: 2 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:46.611868 29853 tablet_service.cc:1514] Processing DeleteTablet for tablet 89938d4c793a44408795398078cc2ee0 with delete_type TABLET_DATA_TOMBSTONED (TS 16440e365430405fb10366ab70e2b429 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58744
I20240503 02:13:46.622548 7406 tablet_replica.cc:330] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:13:46.622807 7406 raft_consensus.cc:2229] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:46.623004 7406 raft_consensus.cc:2258] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:46.623638 7406 ts_tablet_manager.cc:1875] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:46.629945 7406 ts_tablet_manager.cc:1888] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240503 02:13:46.630131 7406 log.cc:1196] T 89938d4c793a44408795398078cc2ee0 P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/89938d4c793a44408795398078cc2ee0
I20240503 02:13:46.630860 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet 89938d4c793a44408795398078cc2ee0 (table rebalancer_extra_table_40 [id=2285665685314d5cae1699d9f2c791f0]) successfully deleted
I20240503 02:13:46.879184 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:46.884526 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:46.910390 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:46.930338 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:46.974792 30244 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:47.012069 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:47.072391 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:47.112154 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:47.208753 30265 consensus_queue.cc:237] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:47.210846 29873 raft_consensus.cc:1261] T f694fefb42d84318a6d91532105b3c90 P 16440e365430405fb10366ab70e2b429 [term 2 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:13:47.211378 566 raft_consensus.cc:1261] T f694fefb42d84318a6d91532105b3c90 P f575ce92c0224eeb80b9fae5f6561fbe [term 2 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:13:47.211565 7394 consensus_queue.cc:1035] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:47.212213 7394 consensus_queue.cc:1035] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:47.214777 7394 raft_consensus.cc:2932] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:47.215117 29873 raft_consensus.cc:2932] T f694fefb42d84318a6d91532105b3c90 P 16440e365430405fb10366ab70e2b429 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:47.223817 29732 catalog_manager.cc:5484] T f694fefb42d84318a6d91532105b3c90 P 16440e365430405fb10366ab70e2b429 reported cstate change: config changed from index 2 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New cstate: current_term: 2 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:47.229430 32755 tablet_service.cc:1514] Processing DeleteTablet for tablet f694fefb42d84318a6d91532105b3c90 with delete_type TABLET_DATA_TOMBSTONED (TS 8a666e798fe8477d8dd1e51b385d5ead not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46360
I20240503 02:13:47.230262 7500 tablet_replica.cc:330] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead: stopping tablet replica
I20240503 02:13:47.230463 7500 raft_consensus.cc:2229] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:47.230631 7500 raft_consensus.cc:2258] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:47.231243 7500 ts_tablet_manager.cc:1875] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:47.234444 566 raft_consensus.cc:2932] T f694fefb42d84318a6d91532105b3c90 P f575ce92c0224eeb80b9fae5f6561fbe [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:47.239356 7500 ts_tablet_manager.cc:1888] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240503 02:13:47.239507 7500 log.cc:1196] T f694fefb42d84318a6d91532105b3c90 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal/wals/f694fefb42d84318a6d91532105b3c90
I20240503 02:13:47.240247 29719 catalog_manager.cc:4830] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): tablet f694fefb42d84318a6d91532105b3c90 (table rebalancer_extra_table_46 [id=95e60300d67c446987cfb1f3dbeecf3a]) successfully deleted
I20240503 02:13:47.273167 566 consensus_queue.cc:237] T c2d33db64ee8482aad840af219d60b8b P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:47.276533 434 raft_consensus.cc:1261] T c2d33db64ee8482aad840af219d60b8b P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:47.277279 7223 consensus_queue.cc:1035] T c2d33db64ee8482aad840af219d60b8b P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:13:47.277549 501 consensus_peers.cc:485] T c2d33db64ee8482aad840af219d60b8b P f575ce92c0224eeb80b9fae5f6561fbe -> Peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Couldn't send request to peer a9094240860e43bfb129193f3bfd5be6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c2d33db64ee8482aad840af219d60b8b. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:47.281926 29873 raft_consensus.cc:1261] T c2d33db64ee8482aad840af219d60b8b P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:47.282366 7223 consensus_queue.cc:1035] T c2d33db64ee8482aad840af219d60b8b P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:47.284790 7517 raft_consensus.cc:2932] T c2d33db64ee8482aad840af219d60b8b P f575ce92c0224eeb80b9fae5f6561fbe [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:47.285435 29873 raft_consensus.cc:2932] T c2d33db64ee8482aad840af219d60b8b P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:47.292336 29732 catalog_manager.cc:5484] T c2d33db64ee8482aad840af219d60b8b P 16440e365430405fb10366ab70e2b429 reported cstate change: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New cstate: current_term: 1 leader_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:47.296720 434 raft_consensus.cc:2932] T c2d33db64ee8482aad840af219d60b8b P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:47.325733 434 consensus_queue.cc:237] T 4cc7861a21f94dabb01133bac9ba0b46 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:47.328128 30004 raft_consensus.cc:1261] T 4cc7861a21f94dabb01133bac9ba0b46 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:13:47.328632 371 consensus_peers.cc:485] T 4cc7861a21f94dabb01133bac9ba0b46 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): Couldn't send request to peer 1f10f80f13c84e80b03eda7085a3b012. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4cc7861a21f94dabb01133bac9ba0b46. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:47.328671 32641 raft_consensus.cc:1261] T 4cc7861a21f94dabb01133bac9ba0b46 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:47.328969 7467 consensus_queue.cc:1035] T 4cc7861a21f94dabb01133bac9ba0b46 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:47.329250 5718 consensus_queue.cc:1035] T 4cc7861a21f94dabb01133bac9ba0b46 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:47.343588 5718 raft_consensus.cc:2932] T 4cc7861a21f94dabb01133bac9ba0b46 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:47.346478 30004 raft_consensus.cc:2932] T 4cc7861a21f94dabb01133bac9ba0b46 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:47.348668 32641 raft_consensus.cc:2932] T 4cc7861a21f94dabb01133bac9ba0b46 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:47.352242 29732 catalog_manager.cc:5484] T 4cc7861a21f94dabb01133bac9ba0b46 P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:47.432650 305 consensus_queue.cc:237] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
W20240503 02:13:47.458441 32709 consensus_peers.cc:485] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Couldn't send request to peer 155735c7f8b14997ac4c770aee2a9401. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac4d497de05c49edbd6e0edac73c058d. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:47.458814 434 raft_consensus.cc:1261] T ac4d497de05c49edbd6e0edac73c058d P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:47.459390 7476 consensus_queue.cc:1035] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:47.459494 566 raft_consensus.cc:1261] T ac4d497de05c49edbd6e0edac73c058d P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:47.460011 7476 consensus_queue.cc:1035] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:47.479995 7123 raft_consensus.cc:2932] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:47.488109 434 raft_consensus.cc:2932] T ac4d497de05c49edbd6e0edac73c058d P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:47.496305 566 raft_consensus.cc:2932] T ac4d497de05c49edbd6e0edac73c058d P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:47.497742 29728 catalog_manager.cc:5484] T ac4d497de05c49edbd6e0edac73c058d P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:47.568770 29873 consensus_queue.cc:237] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:47.571635 30004 raft_consensus.cc:1261] T 46d608d84863424e8c65feac7c2e835f P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:13:47.571882 29808 consensus_peers.cc:485] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 46d608d84863424e8c65feac7c2e835f. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:47.572691 30135 raft_consensus.cc:1261] T 46d608d84863424e8c65feac7c2e835f P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:47.573275 7564 consensus_queue.cc:1035] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:47.573871 7564 consensus_queue.cc:1035] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:47.578892 7576 raft_consensus.cc:2932] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:47.579530 30135 raft_consensus.cc:2932] T 46d608d84863424e8c65feac7c2e835f P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:47.584295 30004 raft_consensus.cc:2932] T 46d608d84863424e8c65feac7c2e835f P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:47.587757 29732 catalog_manager.cc:5484] T 46d608d84863424e8c65feac7c2e835f P a9094240860e43bfb129193f3bfd5be6 reported cstate change: config changed from index -1 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:47.635061 30135 consensus_queue.cc:237] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:47.647763 30004 raft_consensus.cc:1261] T c5cd64614c53487ea1c6cc4a4ca5c379 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:47.648057 29873 raft_consensus.cc:1261] T c5cd64614c53487ea1c6cc4a4ca5c379 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:47.648686 7243 consensus_queue.cc:1035] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:47.650557 7599 consensus_queue.cc:1035] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:13:47.652176 30070 consensus_peers.cc:485] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c5cd64614c53487ea1c6cc4a4ca5c379. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:47.654691 7596 raft_consensus.cc:2932] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:47.655325 30004 raft_consensus.cc:2932] T c5cd64614c53487ea1c6cc4a4ca5c379 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:47.659778 29873 raft_consensus.cc:2932] T c5cd64614c53487ea1c6cc4a4ca5c379 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:47.661808 29726 catalog_manager.cc:5484] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 reported cstate change: config changed from index -1 to 4, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:47.704025 7614 ts_tablet_manager.cc:902] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6: Initiating tablet copy from peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:13:47.705585 7614 tablet_copy_client.cc:308] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.8:42233
I20240503 02:13:47.706389 588 tablet_copy_service.cc:140] P f575ce92c0224eeb80b9fae5f6561fbe: Received BeginTabletCopySession request for tablet c2d33db64ee8482aad840af219d60b8b from peer a9094240860e43bfb129193f3bfd5be6 ({username='slave'} at 127.22.108.3:57589)
I20240503 02:13:47.706585 588 tablet_copy_service.cc:161] P f575ce92c0224eeb80b9fae5f6561fbe: Beginning new tablet copy session on tablet c2d33db64ee8482aad840af219d60b8b from peer a9094240860e43bfb129193f3bfd5be6 at {username='slave'} at 127.22.108.3:57589: session id = a9094240860e43bfb129193f3bfd5be6-c2d33db64ee8482aad840af219d60b8b
I20240503 02:13:47.710907 588 tablet_copy_source_session.cc:205] T c2d33db64ee8482aad840af219d60b8b P f575ce92c0224eeb80b9fae5f6561fbe: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:47.712586 7614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2d33db64ee8482aad840af219d60b8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:47.720579 7614 tablet_copy_client.cc:783] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:47.720821 7614 tablet_copy_client.cc:647] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:47.722504 7614 tablet_copy_client.cc:523] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:47.724884 30004 consensus_queue.cc:237] T 51f5f7297e6448d0b0e9afeedad01429 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:47.725284 7614 tablet_bootstrap.cc:492] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
W20240503 02:13:47.727794 29940 consensus_peers.cc:485] T 51f5f7297e6448d0b0e9afeedad01429 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): Couldn't send request to peer 1f10f80f13c84e80b03eda7085a3b012. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 51f5f7297e6448d0b0e9afeedad01429. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:47.727782 433 raft_consensus.cc:1261] T 51f5f7297e6448d0b0e9afeedad01429 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:47.728009 305 raft_consensus.cc:1261] T 51f5f7297e6448d0b0e9afeedad01429 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:47.728531 7242 consensus_queue.cc:1035] T 51f5f7297e6448d0b0e9afeedad01429 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:47.729090 6638 consensus_queue.cc:1035] T 51f5f7297e6448d0b0e9afeedad01429 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:47.732585 6638 raft_consensus.cc:2932] T 51f5f7297e6448d0b0e9afeedad01429 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:47.736726 7614 tablet_bootstrap.cc:492] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:47.737211 7614 tablet_bootstrap.cc:492] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6: Bootstrap complete.
I20240503 02:13:47.737543 7614 ts_tablet_manager.cc:1367] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.001s
I20240503 02:13:47.737772 433 raft_consensus.cc:2932] T 51f5f7297e6448d0b0e9afeedad01429 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:47.739341 7614 raft_consensus.cc:348] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:47.739554 7614 raft_consensus.cc:729] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:13:47.739802 7614 consensus_queue.cc:260] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:47.740446 7614 ts_tablet_manager.cc:1398] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:13:47.741905 588 tablet_copy_service.cc:342] P f575ce92c0224eeb80b9fae5f6561fbe: Request end of tablet copy session a9094240860e43bfb129193f3bfd5be6-c2d33db64ee8482aad840af219d60b8b received from {username='slave'} at 127.22.108.3:57589
I20240503 02:13:47.742018 588 tablet_copy_service.cc:433] P f575ce92c0224eeb80b9fae5f6561fbe: ending tablet copy session a9094240860e43bfb129193f3bfd5be6-c2d33db64ee8482aad840af219d60b8b on tablet c2d33db64ee8482aad840af219d60b8b with peer a9094240860e43bfb129193f3bfd5be6
I20240503 02:13:47.742344 307 raft_consensus.cc:2932] T 51f5f7297e6448d0b0e9afeedad01429 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:47.748314 29726 catalog_manager.cc:5484] T 51f5f7297e6448d0b0e9afeedad01429 P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:47.783730 433 consensus_queue.cc:237] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:47.786763 567 raft_consensus.cc:1261] T 48b76d7755294d37860842a4a73ab28b P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:13:47.787184 371 consensus_peers.cc:485] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Couldn't send request to peer 155735c7f8b14997ac4c770aee2a9401. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48b76d7755294d37860842a4a73ab28b. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:47.787277 5718 consensus_queue.cc:1035] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:47.790015 30133 raft_consensus.cc:1261] T 48b76d7755294d37860842a4a73ab28b P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:47.790527 7467 consensus_queue.cc:1035] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:47.792922 5718 raft_consensus.cc:2932] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:47.793459 30133 raft_consensus.cc:2932] T 48b76d7755294d37860842a4a73ab28b P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:47.798929 567 raft_consensus.cc:2932] T 48b76d7755294d37860842a4a73ab28b P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:47.799405 29732 catalog_manager.cc:5484] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:47.845525 30265 consensus_queue.cc:237] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
W20240503 02:13:47.848006 30202 consensus_peers.cc:485] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Couldn't send request to peer 16440e365430405fb10366ab70e2b429. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b76b807d2eb24f3ead50bdc0bb1b0231. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:47.848034 567 raft_consensus.cc:1261] T b76b807d2eb24f3ead50bdc0bb1b0231 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:47.848034 30133 raft_consensus.cc:1261] T b76b807d2eb24f3ead50bdc0bb1b0231 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:47.848603 7495 consensus_queue.cc:1035] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:47.850554 7495 consensus_queue.cc:1035] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:47.858513 7495 raft_consensus.cc:2932] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:47.859099 567 raft_consensus.cc:2932] T b76b807d2eb24f3ead50bdc0bb1b0231 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:47.863430 30133 raft_consensus.cc:2932] T b76b807d2eb24f3ead50bdc0bb1b0231 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:47.865912 29732 catalog_manager.cc:5484] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:47.945253 7678 ts_tablet_manager.cc:902] T 4cc7861a21f94dabb01133bac9ba0b46 P 1f10f80f13c84e80b03eda7085a3b012: Initiating tablet copy from peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:48.017158 7684 ts_tablet_manager.cc:902] T ac4d497de05c49edbd6e0edac73c058d P 155735c7f8b14997ac4c770aee2a9401: Initiating tablet copy from peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:48.043550 7678 tablet_copy_client.cc:308] T 4cc7861a21f94dabb01133bac9ba0b46 P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.7:44241
I20240503 02:13:48.044997 457 tablet_copy_service.cc:140] P fbbc2be6382e47908ecb7c68c367bd26: Received BeginTabletCopySession request for tablet 4cc7861a21f94dabb01133bac9ba0b46 from peer 1f10f80f13c84e80b03eda7085a3b012 ({username='slave'} at 127.22.108.4:42187)
I20240503 02:13:48.045193 457 tablet_copy_service.cc:161] P fbbc2be6382e47908ecb7c68c367bd26: Beginning new tablet copy session on tablet 4cc7861a21f94dabb01133bac9ba0b46 from peer 1f10f80f13c84e80b03eda7085a3b012 at {username='slave'} at 127.22.108.4:42187: session id = 1f10f80f13c84e80b03eda7085a3b012-4cc7861a21f94dabb01133bac9ba0b46
I20240503 02:13:48.048257 457 tablet_copy_source_session.cc:205] T 4cc7861a21f94dabb01133bac9ba0b46 P fbbc2be6382e47908ecb7c68c367bd26: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:48.049796 7678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cc7861a21f94dabb01133bac9ba0b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:48.058313 7678 tablet_copy_client.cc:783] T 4cc7861a21f94dabb01133bac9ba0b46 P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:48.058660 7678 tablet_copy_client.cc:647] T 4cc7861a21f94dabb01133bac9ba0b46 P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:48.060539 7678 tablet_copy_client.cc:523] T 4cc7861a21f94dabb01133bac9ba0b46 P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:48.074116 7678 tablet_bootstrap.cc:492] T 4cc7861a21f94dabb01133bac9ba0b46 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:13:48.077376 7684 tablet_copy_client.cc:308] T ac4d497de05c49edbd6e0edac73c058d P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.6:39975
I20240503 02:13:48.087035 327 tablet_copy_service.cc:140] P 8a666e798fe8477d8dd1e51b385d5ead: Received BeginTabletCopySession request for tablet ac4d497de05c49edbd6e0edac73c058d from peer 155735c7f8b14997ac4c770aee2a9401 ({username='slave'} at 127.22.108.2:53863)
I20240503 02:13:48.087260 327 tablet_copy_service.cc:161] P 8a666e798fe8477d8dd1e51b385d5ead: Beginning new tablet copy session on tablet ac4d497de05c49edbd6e0edac73c058d from peer 155735c7f8b14997ac4c770aee2a9401 at {username='slave'} at 127.22.108.2:53863: session id = 155735c7f8b14997ac4c770aee2a9401-ac4d497de05c49edbd6e0edac73c058d
I20240503 02:13:48.091243 327 tablet_copy_source_session.cc:205] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:48.095907 7684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac4d497de05c49edbd6e0edac73c058d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:48.105574 7684 tablet_copy_client.cc:783] T ac4d497de05c49edbd6e0edac73c058d P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:48.105796 7684 tablet_copy_client.cc:647] T ac4d497de05c49edbd6e0edac73c058d P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:48.108256 7678 tablet_bootstrap.cc:492] T 4cc7861a21f94dabb01133bac9ba0b46 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:48.108726 7678 tablet_bootstrap.cc:492] T 4cc7861a21f94dabb01133bac9ba0b46 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap complete.
I20240503 02:13:48.109076 7678 ts_tablet_manager.cc:1367] T 4cc7861a21f94dabb01133bac9ba0b46 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.035s user 0.009s sys 0.010s
I20240503 02:13:48.110149 7694 ts_tablet_manager.cc:902] T 46d608d84863424e8c65feac7c2e835f P f575ce92c0224eeb80b9fae5f6561fbe: Initiating tablet copy from peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:13:48.110157 7678 raft_consensus.cc:348] T 4cc7861a21f94dabb01133bac9ba0b46 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:48.110392 7678 raft_consensus.cc:729] T 4cc7861a21f94dabb01133bac9ba0b46 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:13:48.110677 7684 tablet_copy_client.cc:523] T ac4d497de05c49edbd6e0edac73c058d P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:48.110666 7678 consensus_queue.cc:260] T 4cc7861a21f94dabb01133bac9ba0b46 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:48.114136 7684 tablet_bootstrap.cc:492] T ac4d497de05c49edbd6e0edac73c058d P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:13:48.119014 7678 ts_tablet_manager.cc:1398] T 4cc7861a21f94dabb01133bac9ba0b46 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 02:13:48.119804 7684 tablet_bootstrap.cc:492] T ac4d497de05c49edbd6e0edac73c058d P 155735c7f8b14997ac4c770aee2a9401: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:48.119823 457 tablet_copy_service.cc:342] P fbbc2be6382e47908ecb7c68c367bd26: Request end of tablet copy session 1f10f80f13c84e80b03eda7085a3b012-4cc7861a21f94dabb01133bac9ba0b46 received from {username='slave'} at 127.22.108.4:42187
I20240503 02:13:48.119969 457 tablet_copy_service.cc:433] P fbbc2be6382e47908ecb7c68c367bd26: ending tablet copy session 1f10f80f13c84e80b03eda7085a3b012-4cc7861a21f94dabb01133bac9ba0b46 on tablet 4cc7861a21f94dabb01133bac9ba0b46 with peer 1f10f80f13c84e80b03eda7085a3b012
I20240503 02:13:48.120143 7684 tablet_bootstrap.cc:492] T ac4d497de05c49edbd6e0edac73c058d P 155735c7f8b14997ac4c770aee2a9401: Bootstrap complete.
I20240503 02:13:48.120335 7684 ts_tablet_manager.cc:1367] T ac4d497de05c49edbd6e0edac73c058d P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 02:13:48.121202 7684 raft_consensus.cc:348] T ac4d497de05c49edbd6e0edac73c058d P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:48.121438 7684 raft_consensus.cc:729] T ac4d497de05c49edbd6e0edac73c058d P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:13:48.122619 29873 consensus_queue.cc:237] T c97c7a843366426887115e13f35faf52 P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:48.122457 7684 consensus_queue.cc:260] T ac4d497de05c49edbd6e0edac73c058d P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:48.122921 7694 tablet_copy_client.cc:308] T 46d608d84863424e8c65feac7c2e835f P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.1:36719
I20240503 02:13:48.123476 7684 ts_tablet_manager.cc:1398] T ac4d497de05c49edbd6e0edac73c058d P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20240503 02:13:48.123555 29895 tablet_copy_service.cc:140] P 16440e365430405fb10366ab70e2b429: Received BeginTabletCopySession request for tablet 46d608d84863424e8c65feac7c2e835f from peer f575ce92c0224eeb80b9fae5f6561fbe ({username='slave'} at 127.22.108.8:43859)
I20240503 02:13:48.123679 29895 tablet_copy_service.cc:161] P 16440e365430405fb10366ab70e2b429: Beginning new tablet copy session on tablet 46d608d84863424e8c65feac7c2e835f from peer f575ce92c0224eeb80b9fae5f6561fbe at {username='slave'} at 127.22.108.8:43859: session id = f575ce92c0224eeb80b9fae5f6561fbe-46d608d84863424e8c65feac7c2e835f
I20240503 02:13:48.126292 307 raft_consensus.cc:1261] T c97c7a843366426887115e13f35faf52 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:48.126322 29895 tablet_copy_source_session.cc:205] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:48.127089 327 tablet_copy_service.cc:342] P 8a666e798fe8477d8dd1e51b385d5ead: Request end of tablet copy session 155735c7f8b14997ac4c770aee2a9401-ac4d497de05c49edbd6e0edac73c058d received from {username='slave'} at 127.22.108.2:53863
I20240503 02:13:48.127189 327 tablet_copy_service.cc:433] P 8a666e798fe8477d8dd1e51b385d5ead: ending tablet copy session 155735c7f8b14997ac4c770aee2a9401-ac4d497de05c49edbd6e0edac73c058d on tablet ac4d497de05c49edbd6e0edac73c058d with peer 155735c7f8b14997ac4c770aee2a9401
I20240503 02:13:48.127691 32640 raft_consensus.cc:1261] T c97c7a843366426887115e13f35faf52 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:48.128000 7694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d608d84863424e8c65feac7c2e835f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:48.128310 7579 consensus_queue.cc:1035] T c97c7a843366426887115e13f35faf52 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:13:48.128832 29809 consensus_peers.cc:485] T c97c7a843366426887115e13f35faf52 P 16440e365430405fb10366ab70e2b429 -> Peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): Couldn't send request to peer fbbc2be6382e47908ecb7c68c367bd26. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c97c7a843366426887115e13f35faf52. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:48.128841 7579 consensus_queue.cc:1035] T c97c7a843366426887115e13f35faf52 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 02:13:48.141072 7694 tablet_copy_client.cc:783] T 46d608d84863424e8c65feac7c2e835f P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:48.141315 7694 tablet_copy_client.cc:647] T 46d608d84863424e8c65feac7c2e835f P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:48.141533 30133 raft_consensus.cc:1203] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:48.150899 7694 tablet_copy_client.cc:523] T 46d608d84863424e8c65feac7c2e835f P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:48.153123 7576 raft_consensus.cc:2932] T c97c7a843366426887115e13f35faf52 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:48.155357 7694 tablet_bootstrap.cc:492] T 46d608d84863424e8c65feac7c2e835f P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:13:48.159345 306 raft_consensus.cc:2932] T c97c7a843366426887115e13f35faf52 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:48.167007 32641 raft_consensus.cc:2932] T c97c7a843366426887115e13f35faf52 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:48.174510 29728 catalog_manager.cc:5484] T c97c7a843366426887115e13f35faf52 P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: config changed from index -1 to 2, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:48.178911 7705 ts_tablet_manager.cc:902] T c5cd64614c53487ea1c6cc4a4ca5c379 P 9a4d92bc25374fb3960fde7d10be309d: Initiating tablet copy from peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:13:48.190390 7678 ts_tablet_manager.cc:902] T 51f5f7297e6448d0b0e9afeedad01429 P 1f10f80f13c84e80b03eda7085a3b012: Initiating tablet copy from peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:13:48.208792 7705 tablet_copy_client.cc:308] T c5cd64614c53487ea1c6cc4a4ca5c379 P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.3:42679
I20240503 02:13:48.212004 30156 tablet_copy_service.cc:140] P a9094240860e43bfb129193f3bfd5be6: Received BeginTabletCopySession request for tablet c5cd64614c53487ea1c6cc4a4ca5c379 from peer 9a4d92bc25374fb3960fde7d10be309d ({username='slave'} at 127.22.108.5:33825)
I20240503 02:13:48.212239 30156 tablet_copy_service.cc:161] P a9094240860e43bfb129193f3bfd5be6: Beginning new tablet copy session on tablet c5cd64614c53487ea1c6cc4a4ca5c379 from peer 9a4d92bc25374fb3960fde7d10be309d at {username='slave'} at 127.22.108.5:33825: session id = 9a4d92bc25374fb3960fde7d10be309d-c5cd64614c53487ea1c6cc4a4ca5c379
I20240503 02:13:48.215360 30156 tablet_copy_source_session.cc:205] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:48.216907 7705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5cd64614c53487ea1c6cc4a4ca5c379. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:48.233690 7684 ts_tablet_manager.cc:902] T 48b76d7755294d37860842a4a73ab28b P 155735c7f8b14997ac4c770aee2a9401: Initiating tablet copy from peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:48.247741 7705 tablet_copy_client.cc:783] T c5cd64614c53487ea1c6cc4a4ca5c379 P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:48.248078 7705 tablet_copy_client.cc:647] T c5cd64614c53487ea1c6cc4a4ca5c379 P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:48.253947 7705 tablet_copy_client.cc:523] T c5cd64614c53487ea1c6cc4a4ca5c379 P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:48.258342 7705 tablet_bootstrap.cc:492] T c5cd64614c53487ea1c6cc4a4ca5c379 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:48.258368 7694 tablet_bootstrap.cc:492] T 46d608d84863424e8c65feac7c2e835f P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:48.258819 7694 tablet_bootstrap.cc:492] T 46d608d84863424e8c65feac7c2e835f P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap complete.
I20240503 02:13:48.259101 7694 ts_tablet_manager.cc:1367] T 46d608d84863424e8c65feac7c2e835f P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.104s user 0.063s sys 0.016s
I20240503 02:13:48.259938 7694 raft_consensus.cc:348] T 46d608d84863424e8c65feac7c2e835f P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:48.260218 7694 raft_consensus.cc:729] T 46d608d84863424e8c65feac7c2e835f P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:13:48.260460 7694 consensus_queue.cc:260] T 46d608d84863424e8c65feac7c2e835f P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:48.261276 7694 ts_tablet_manager.cc:1398] T 46d608d84863424e8c65feac7c2e835f P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20240503 02:13:48.261678 7678 tablet_copy_client.cc:308] T 51f5f7297e6448d0b0e9afeedad01429 P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.2:37873
I20240503 02:13:48.262197 29895 tablet_copy_service.cc:342] P 16440e365430405fb10366ab70e2b429: Request end of tablet copy session f575ce92c0224eeb80b9fae5f6561fbe-46d608d84863424e8c65feac7c2e835f received from {username='slave'} at 127.22.108.8:43859
I20240503 02:13:48.262306 29895 tablet_copy_service.cc:433] P 16440e365430405fb10366ab70e2b429: ending tablet copy session f575ce92c0224eeb80b9fae5f6561fbe-46d608d84863424e8c65feac7c2e835f on tablet 46d608d84863424e8c65feac7c2e835f with peer f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:13:48.262488 30026 tablet_copy_service.cc:140] P 155735c7f8b14997ac4c770aee2a9401: Received BeginTabletCopySession request for tablet 51f5f7297e6448d0b0e9afeedad01429 from peer 1f10f80f13c84e80b03eda7085a3b012 ({username='slave'} at 127.22.108.4:56219)
I20240503 02:13:48.262615 30026 tablet_copy_service.cc:161] P 155735c7f8b14997ac4c770aee2a9401: Beginning new tablet copy session on tablet 51f5f7297e6448d0b0e9afeedad01429 from peer 1f10f80f13c84e80b03eda7085a3b012 at {username='slave'} at 127.22.108.4:56219: session id = 1f10f80f13c84e80b03eda7085a3b012-51f5f7297e6448d0b0e9afeedad01429
I20240503 02:13:48.266278 30026 tablet_copy_source_session.cc:205] T 51f5f7297e6448d0b0e9afeedad01429 P 155735c7f8b14997ac4c770aee2a9401: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:48.270138 7678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51f5f7297e6448d0b0e9afeedad01429. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:48.279249 7678 tablet_copy_client.cc:783] T 51f5f7297e6448d0b0e9afeedad01429 P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:48.279991 7678 tablet_copy_client.cc:647] T 51f5f7297e6448d0b0e9afeedad01429 P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:48.282232 7678 tablet_copy_client.cc:523] T 51f5f7297e6448d0b0e9afeedad01429 P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:48.292783 7678 tablet_bootstrap.cc:492] T 51f5f7297e6448d0b0e9afeedad01429 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:13:48.295738 7684 tablet_copy_client.cc:308] T 48b76d7755294d37860842a4a73ab28b P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.7:44241
I20240503 02:13:48.297529 457 tablet_copy_service.cc:140] P fbbc2be6382e47908ecb7c68c367bd26: Received BeginTabletCopySession request for tablet 48b76d7755294d37860842a4a73ab28b from peer 155735c7f8b14997ac4c770aee2a9401 ({username='slave'} at 127.22.108.2:48617)
I20240503 02:13:48.297683 457 tablet_copy_service.cc:161] P fbbc2be6382e47908ecb7c68c367bd26: Beginning new tablet copy session on tablet 48b76d7755294d37860842a4a73ab28b from peer 155735c7f8b14997ac4c770aee2a9401 at {username='slave'} at 127.22.108.2:48617: session id = 155735c7f8b14997ac4c770aee2a9401-48b76d7755294d37860842a4a73ab28b
I20240503 02:13:48.298496 7731 ts_tablet_manager.cc:902] T b76b807d2eb24f3ead50bdc0bb1b0231 P 16440e365430405fb10366ab70e2b429: Initiating tablet copy from peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:13:48.300405 457 tablet_copy_source_session.cc:205] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:48.305271 7684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48b76d7755294d37860842a4a73ab28b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:48.314518 7684 tablet_copy_client.cc:783] T 48b76d7755294d37860842a4a73ab28b P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:48.314823 7684 tablet_copy_client.cc:647] T 48b76d7755294d37860842a4a73ab28b P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:48.315608 7678 tablet_bootstrap.cc:492] T 51f5f7297e6448d0b0e9afeedad01429 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:48.316031 7678 tablet_bootstrap.cc:492] T 51f5f7297e6448d0b0e9afeedad01429 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap complete.
I20240503 02:13:48.316331 7678 ts_tablet_manager.cc:1367] T 51f5f7297e6448d0b0e9afeedad01429 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.005s
I20240503 02:13:48.317467 7678 raft_consensus.cc:348] T 51f5f7297e6448d0b0e9afeedad01429 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:48.317796 7678 raft_consensus.cc:729] T 51f5f7297e6448d0b0e9afeedad01429 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:13:48.318132 7678 consensus_queue.cc:260] T 51f5f7297e6448d0b0e9afeedad01429 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:48.318981 7678 ts_tablet_manager.cc:1398] T 51f5f7297e6448d0b0e9afeedad01429 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:13:48.319787 30026 tablet_copy_service.cc:342] P 155735c7f8b14997ac4c770aee2a9401: Request end of tablet copy session 1f10f80f13c84e80b03eda7085a3b012-51f5f7297e6448d0b0e9afeedad01429 received from {username='slave'} at 127.22.108.4:56219
I20240503 02:13:48.319886 30026 tablet_copy_service.cc:433] P 155735c7f8b14997ac4c770aee2a9401: ending tablet copy session 1f10f80f13c84e80b03eda7085a3b012-51f5f7297e6448d0b0e9afeedad01429 on tablet 51f5f7297e6448d0b0e9afeedad01429 with peer 1f10f80f13c84e80b03eda7085a3b012
I20240503 02:13:48.328009 7731 tablet_copy_client.cc:308] T b76b807d2eb24f3ead50bdc0bb1b0231 P 16440e365430405fb10366ab70e2b429: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.4:33935
I20240503 02:13:48.328667 7684 tablet_copy_client.cc:523] T 48b76d7755294d37860842a4a73ab28b P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:48.328727 30286 tablet_copy_service.cc:140] P 1f10f80f13c84e80b03eda7085a3b012: Received BeginTabletCopySession request for tablet b76b807d2eb24f3ead50bdc0bb1b0231 from peer 16440e365430405fb10366ab70e2b429 ({username='slave'} at 127.22.108.1:38817)
I20240503 02:13:48.328881 30286 tablet_copy_service.cc:161] P 1f10f80f13c84e80b03eda7085a3b012: Beginning new tablet copy session on tablet b76b807d2eb24f3ead50bdc0bb1b0231 from peer 16440e365430405fb10366ab70e2b429 at {username='slave'} at 127.22.108.1:38817: session id = 16440e365430405fb10366ab70e2b429-b76b807d2eb24f3ead50bdc0bb1b0231
I20240503 02:13:48.332022 7684 tablet_bootstrap.cc:492] T 48b76d7755294d37860842a4a73ab28b P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:13:48.332711 30286 tablet_copy_source_session.cc:205] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:48.334322 7731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b76b807d2eb24f3ead50bdc0bb1b0231. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:48.348027 7731 tablet_copy_client.cc:783] T b76b807d2eb24f3ead50bdc0bb1b0231 P 16440e365430405fb10366ab70e2b429: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:48.348416 7731 tablet_copy_client.cc:647] T b76b807d2eb24f3ead50bdc0bb1b0231 P 16440e365430405fb10366ab70e2b429: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:48.350554 7731 tablet_copy_client.cc:523] T b76b807d2eb24f3ead50bdc0bb1b0231 P 16440e365430405fb10366ab70e2b429: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:48.363579 7705 tablet_bootstrap.cc:492] T c5cd64614c53487ea1c6cc4a4ca5c379 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:48.364073 7705 tablet_bootstrap.cc:492] T c5cd64614c53487ea1c6cc4a4ca5c379 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap complete.
I20240503 02:13:48.364353 7705 ts_tablet_manager.cc:1367] T c5cd64614c53487ea1c6cc4a4ca5c379 P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.106s user 0.059s sys 0.000s
I20240503 02:13:48.365229 7705 raft_consensus.cc:348] T c5cd64614c53487ea1c6cc4a4ca5c379 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:48.365459 7705 raft_consensus.cc:729] T c5cd64614c53487ea1c6cc4a4ca5c379 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:48.365715 7705 consensus_queue.cc:260] T c5cd64614c53487ea1c6cc4a4ca5c379 P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:48.366266 7705 ts_tablet_manager.cc:1398] T c5cd64614c53487ea1c6cc4a4ca5c379 P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:48.367606 30156 tablet_copy_service.cc:342] P a9094240860e43bfb129193f3bfd5be6: Request end of tablet copy session 9a4d92bc25374fb3960fde7d10be309d-c5cd64614c53487ea1c6cc4a4ca5c379 received from {username='slave'} at 127.22.108.5:33825
I20240503 02:13:48.367723 30156 tablet_copy_service.cc:433] P a9094240860e43bfb129193f3bfd5be6: ending tablet copy session 9a4d92bc25374fb3960fde7d10be309d-c5cd64614c53487ea1c6cc4a4ca5c379 on tablet c5cd64614c53487ea1c6cc4a4ca5c379 with peer 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:13:48.380298 7731 tablet_bootstrap.cc:492] T b76b807d2eb24f3ead50bdc0bb1b0231 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:13:48.409811 7684 tablet_bootstrap.cc:492] T 48b76d7755294d37860842a4a73ab28b P 155735c7f8b14997ac4c770aee2a9401: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:48.410281 7684 tablet_bootstrap.cc:492] T 48b76d7755294d37860842a4a73ab28b P 155735c7f8b14997ac4c770aee2a9401: Bootstrap complete.
I20240503 02:13:48.410516 7684 ts_tablet_manager.cc:1367] T 48b76d7755294d37860842a4a73ab28b P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.079s user 0.013s sys 0.023s
I20240503 02:13:48.411893 433 consensus_queue.cc:237] T 9c6e204df01e450598b05c3429574700 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:48.411310 7684 raft_consensus.cc:348] T 48b76d7755294d37860842a4a73ab28b P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:48.415809 7684 raft_consensus.cc:729] T 48b76d7755294d37860842a4a73ab28b P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:13:48.416642 7684 consensus_queue.cc:260] T 48b76d7755294d37860842a4a73ab28b P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:48.418067 7684 ts_tablet_manager.cc:1398] T 48b76d7755294d37860842a4a73ab28b P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:13:48.420096 457 tablet_copy_service.cc:342] P fbbc2be6382e47908ecb7c68c367bd26: Request end of tablet copy session 155735c7f8b14997ac4c770aee2a9401-48b76d7755294d37860842a4a73ab28b received from {username='slave'} at 127.22.108.2:48617
I20240503 02:13:48.420257 457 tablet_copy_service.cc:433] P fbbc2be6382e47908ecb7c68c367bd26: ending tablet copy session 155735c7f8b14997ac4c770aee2a9401-48b76d7755294d37860842a4a73ab28b on tablet 48b76d7755294d37860842a4a73ab28b with peer 155735c7f8b14997ac4c770aee2a9401
I20240503 02:13:48.424777 567 raft_consensus.cc:1261] T 9c6e204df01e450598b05c3429574700 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:48.425864 5718 consensus_queue.cc:1035] T 9c6e204df01e450598b05c3429574700 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:13:48.426544 371 consensus_peers.cc:485] T 9c6e204df01e450598b05c3429574700 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): Couldn't send request to peer 1f10f80f13c84e80b03eda7085a3b012. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9c6e204df01e450598b05c3429574700. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:48.433715 307 raft_consensus.cc:1261] T 9c6e204df01e450598b05c3429574700 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:48.434913 5718 consensus_queue.cc:1035] T 9c6e204df01e450598b05c3429574700 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:48.437752 30265 raft_consensus.cc:1203] T 4cc7861a21f94dabb01133bac9ba0b46 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:48.441289 7731 tablet_bootstrap.cc:492] T b76b807d2eb24f3ead50bdc0bb1b0231 P 16440e365430405fb10366ab70e2b429: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:48.441815 7731 tablet_bootstrap.cc:492] T b76b807d2eb24f3ead50bdc0bb1b0231 P 16440e365430405fb10366ab70e2b429: Bootstrap complete.
I20240503 02:13:48.442224 7731 ts_tablet_manager.cc:1367] T b76b807d2eb24f3ead50bdc0bb1b0231 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.062s user 0.010s sys 0.009s
I20240503 02:13:48.443029 7731 raft_consensus.cc:348] T b76b807d2eb24f3ead50bdc0bb1b0231 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:48.443312 7731 raft_consensus.cc:729] T b76b807d2eb24f3ead50bdc0bb1b0231 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:13:48.443596 7731 consensus_queue.cc:260] T b76b807d2eb24f3ead50bdc0bb1b0231 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:48.444386 7731 ts_tablet_manager.cc:1398] T b76b807d2eb24f3ead50bdc0bb1b0231 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:13:48.445201 30286 tablet_copy_service.cc:342] P 1f10f80f13c84e80b03eda7085a3b012: Request end of tablet copy session 16440e365430405fb10366ab70e2b429-b76b807d2eb24f3ead50bdc0bb1b0231 received from {username='slave'} at 127.22.108.1:38817
I20240503 02:13:48.445333 30286 tablet_copy_service.cc:433] P 1f10f80f13c84e80b03eda7085a3b012: ending tablet copy session 16440e365430405fb10366ab70e2b429-b76b807d2eb24f3ead50bdc0bb1b0231 on tablet b76b807d2eb24f3ead50bdc0bb1b0231 with peer 16440e365430405fb10366ab70e2b429
I20240503 02:13:48.454851 7467 raft_consensus.cc:2932] T 9c6e204df01e450598b05c3429574700 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:48.459740 307 raft_consensus.cc:2932] T 9c6e204df01e450598b05c3429574700 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:48.463099 567 raft_consensus.cc:2932] T 9c6e204df01e450598b05c3429574700 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:48.464637 29728 catalog_manager.cc:5484] T 9c6e204df01e450598b05c3429574700 P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:48.499439 567 raft_consensus.cc:1203] T 46d608d84863424e8c65feac7c2e835f P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:48.505328 30004 raft_consensus.cc:1203] T ac4d497de05c49edbd6e0edac73c058d P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:48.561547 7760 ts_tablet_manager.cc:902] T c97c7a843366426887115e13f35faf52 P fbbc2be6382e47908ecb7c68c367bd26: Initiating tablet copy from peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:13:48.565349 7760 tablet_copy_client.cc:308] T c97c7a843366426887115e13f35faf52 P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.1:36719
I20240503 02:13:48.565999 29895 tablet_copy_service.cc:140] P 16440e365430405fb10366ab70e2b429: Received BeginTabletCopySession request for tablet c97c7a843366426887115e13f35faf52 from peer fbbc2be6382e47908ecb7c68c367bd26 ({username='slave'} at 127.22.108.7:58281)
I20240503 02:13:48.566121 29895 tablet_copy_service.cc:161] P 16440e365430405fb10366ab70e2b429: Beginning new tablet copy session on tablet c97c7a843366426887115e13f35faf52 from peer fbbc2be6382e47908ecb7c68c367bd26 at {username='slave'} at 127.22.108.7:58281: session id = fbbc2be6382e47908ecb7c68c367bd26-c97c7a843366426887115e13f35faf52
I20240503 02:13:48.569594 29895 tablet_copy_source_session.cc:205] T c97c7a843366426887115e13f35faf52 P 16440e365430405fb10366ab70e2b429: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:48.571136 7760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c97c7a843366426887115e13f35faf52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:48.573673 30006 consensus_queue.cc:237] T b7a2201f9605454ea12df4e3a7e5a63e P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:48.577132 7760 tablet_copy_client.cc:783] T c97c7a843366426887115e13f35faf52 P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:48.577245 30135 raft_consensus.cc:1261] T b7a2201f9605454ea12df4e3a7e5a63e P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:48.577273 29873 raft_consensus.cc:1261] T b7a2201f9605454ea12df4e3a7e5a63e P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:48.577342 7760 tablet_copy_client.cc:647] T c97c7a843366426887115e13f35faf52 P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Starting download of 1 WAL segments...
W20240503 02:13:48.578392 29940 consensus_peers.cc:485] T b7a2201f9605454ea12df4e3a7e5a63e P 155735c7f8b14997ac4c770aee2a9401 -> Peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): Couldn't send request to peer 8a666e798fe8477d8dd1e51b385d5ead. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b7a2201f9605454ea12df4e3a7e5a63e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:48.578598 7628 consensus_queue.cc:1035] T b7a2201f9605454ea12df4e3a7e5a63e P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:48.579293 7760 tablet_copy_client.cc:523] T c97c7a843366426887115e13f35faf52 P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:48.580476 7624 consensus_queue.cc:1035] T b7a2201f9605454ea12df4e3a7e5a63e P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:48.582528 7760 tablet_bootstrap.cc:492] T c97c7a843366426887115e13f35faf52 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:13:48.583236 7771 raft_consensus.cc:2932] T b7a2201f9605454ea12df4e3a7e5a63e P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:48.585548 29873 raft_consensus.cc:2932] T b7a2201f9605454ea12df4e3a7e5a63e P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:48.588438 7760 tablet_bootstrap.cc:492] T c97c7a843366426887115e13f35faf52 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:48.588886 7760 tablet_bootstrap.cc:492] T c97c7a843366426887115e13f35faf52 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap complete.
I20240503 02:13:48.589203 7760 ts_tablet_manager.cc:1367] T c97c7a843366426887115e13f35faf52 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20240503 02:13:48.589924 7760 raft_consensus.cc:348] T c97c7a843366426887115e13f35faf52 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:48.590204 7760 raft_consensus.cc:729] T c97c7a843366426887115e13f35faf52 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:13:48.590420 7760 consensus_queue.cc:260] T c97c7a843366426887115e13f35faf52 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:48.590991 7760 ts_tablet_manager.cc:1398] T c97c7a843366426887115e13f35faf52 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:13:48.592463 29895 tablet_copy_service.cc:342] P 16440e365430405fb10366ab70e2b429: Request end of tablet copy session fbbc2be6382e47908ecb7c68c367bd26-c97c7a843366426887115e13f35faf52 received from {username='slave'} at 127.22.108.7:58281
I20240503 02:13:48.592569 29895 tablet_copy_service.cc:433] P 16440e365430405fb10366ab70e2b429: ending tablet copy session fbbc2be6382e47908ecb7c68c367bd26-c97c7a843366426887115e13f35faf52 on tablet c97c7a843366426887115e13f35faf52 with peer fbbc2be6382e47908ecb7c68c367bd26
I20240503 02:13:48.592888 29726 catalog_manager.cc:5484] T b7a2201f9605454ea12df4e3a7e5a63e P 16440e365430405fb10366ab70e2b429 reported cstate change: config changed from index -1 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:48.600069 30135 raft_consensus.cc:2932] T b7a2201f9605454ea12df4e3a7e5a63e P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:48.656214 307 consensus_queue.cc:237] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:48.658535 567 raft_consensus.cc:1261] T eb70e12d4fdb4d15a93d37dfdc659a14 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:48.658627 433 raft_consensus.cc:1261] T eb70e12d4fdb4d15a93d37dfdc659a14 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:48.659137 7476 consensus_queue.cc:1035] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:13:48.659466 32709 consensus_peers.cc:485] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Couldn't send request to peer 155735c7f8b14997ac4c770aee2a9401. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb70e12d4fdb4d15a93d37dfdc659a14. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:48.660100 7476 consensus_queue.cc:1035] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:48.662547 32641 raft_consensus.cc:1203] T c5cd64614c53487ea1c6cc4a4ca5c379 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:48.663015 7476 raft_consensus.cc:2932] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:48.663450 567 raft_consensus.cc:2932] T eb70e12d4fdb4d15a93d37dfdc659a14 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:48.667872 30266 raft_consensus.cc:1203] T 51f5f7297e6448d0b0e9afeedad01429 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:48.671471 29732 catalog_manager.cc:5484] T eb70e12d4fdb4d15a93d37dfdc659a14 P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:48.675405 433 raft_consensus.cc:2932] T eb70e12d4fdb4d15a93d37dfdc659a14 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:48.713876 30006 raft_consensus.cc:1203] T 48b76d7755294d37860842a4a73ab28b P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:48.723134 433 consensus_queue.cc:237] T 5ccde7688c4e49b5b9d89d6e048c308e P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:48.725301 32641 raft_consensus.cc:1261] T 5ccde7688c4e49b5b9d89d6e048c308e P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:48.725340 306 raft_consensus.cc:1261] T 5ccde7688c4e49b5b9d89d6e048c308e P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:13:48.725369 373 consensus_peers.cc:485] T 5ccde7688c4e49b5b9d89d6e048c308e P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Couldn't send request to peer 16440e365430405fb10366ab70e2b429. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ccde7688c4e49b5b9d89d6e048c308e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:48.725843 6997 consensus_queue.cc:1035] T 5ccde7688c4e49b5b9d89d6e048c308e P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:48.726112 7467 consensus_queue.cc:1035] T 5ccde7688c4e49b5b9d89d6e048c308e P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:48.728554 6997 raft_consensus.cc:2932] T 5ccde7688c4e49b5b9d89d6e048c308e P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:48.729151 32641 raft_consensus.cc:2932] T 5ccde7688c4e49b5b9d89d6e048c308e P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:48.736366 29732 catalog_manager.cc:5484] T 5ccde7688c4e49b5b9d89d6e048c308e P 9a4d92bc25374fb3960fde7d10be309d reported cstate change: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:48.740139 306 raft_consensus.cc:2932] T 5ccde7688c4e49b5b9d89d6e048c308e P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:48.802902 29873 raft_consensus.cc:1203] T b76b807d2eb24f3ead50bdc0bb1b0231 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:48.942343 7853 ts_tablet_manager.cc:902] T 9c6e204df01e450598b05c3429574700 P 1f10f80f13c84e80b03eda7085a3b012: Initiating tablet copy from peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:48.961607 7853 tablet_copy_client.cc:308] T 9c6e204df01e450598b05c3429574700 P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.7:44241
I20240503 02:13:48.962234 457 tablet_copy_service.cc:140] P fbbc2be6382e47908ecb7c68c367bd26: Received BeginTabletCopySession request for tablet 9c6e204df01e450598b05c3429574700 from peer 1f10f80f13c84e80b03eda7085a3b012 ({username='slave'} at 127.22.108.4:42187)
I20240503 02:13:48.962410 457 tablet_copy_service.cc:161] P fbbc2be6382e47908ecb7c68c367bd26: Beginning new tablet copy session on tablet 9c6e204df01e450598b05c3429574700 from peer 1f10f80f13c84e80b03eda7085a3b012 at {username='slave'} at 127.22.108.4:42187: session id = 1f10f80f13c84e80b03eda7085a3b012-9c6e204df01e450598b05c3429574700
I20240503 02:13:48.965209 457 tablet_copy_source_session.cc:205] T 9c6e204df01e450598b05c3429574700 P fbbc2be6382e47908ecb7c68c367bd26: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:48.985482 434 raft_consensus.cc:1203] T c97c7a843366426887115e13f35faf52 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:48.989822 7853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c6e204df01e450598b05c3429574700. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:48.997767 7853 tablet_copy_client.cc:783] T 9c6e204df01e450598b05c3429574700 P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:48.998093 7853 tablet_copy_client.cc:647] T 9c6e204df01e450598b05c3429574700 P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:49.003713 7853 tablet_copy_client.cc:523] T 9c6e204df01e450598b05c3429574700 P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:49.009246 7853 tablet_bootstrap.cc:492] T 9c6e204df01e450598b05c3429574700 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:13:49.031977 7853 tablet_bootstrap.cc:492] T 9c6e204df01e450598b05c3429574700 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:49.032500 7853 tablet_bootstrap.cc:492] T 9c6e204df01e450598b05c3429574700 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap complete.
I20240503 02:13:49.032822 7853 ts_tablet_manager.cc:1367] T 9c6e204df01e450598b05c3429574700 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.000s
I20240503 02:13:49.033634 7853 raft_consensus.cc:348] T 9c6e204df01e450598b05c3429574700 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:49.033989 7853 raft_consensus.cc:729] T 9c6e204df01e450598b05c3429574700 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:13:49.034255 7853 consensus_queue.cc:260] T 9c6e204df01e450598b05c3429574700 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:49.038420 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:49.042917 7853 ts_tablet_manager.cc:1398] T 9c6e204df01e450598b05c3429574700 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 02:13:49.045137 457 tablet_copy_service.cc:342] P fbbc2be6382e47908ecb7c68c367bd26: Request end of tablet copy session 1f10f80f13c84e80b03eda7085a3b012-9c6e204df01e450598b05c3429574700 received from {username='slave'} at 127.22.108.4:42187
I20240503 02:13:49.045258 457 tablet_copy_service.cc:433] P fbbc2be6382e47908ecb7c68c367bd26: ending tablet copy session 1f10f80f13c84e80b03eda7085a3b012-9c6e204df01e450598b05c3429574700 on tablet 9c6e204df01e450598b05c3429574700 with peer 1f10f80f13c84e80b03eda7085a3b012
I20240503 02:13:49.079843 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:49.090521 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:49.110869 30244 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:49.148073 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:49.158011 7868 ts_tablet_manager.cc:902] T eb70e12d4fdb4d15a93d37dfdc659a14 P 155735c7f8b14997ac4c770aee2a9401: Initiating tablet copy from peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:49.160429 7873 ts_tablet_manager.cc:902] T 5ccde7688c4e49b5b9d89d6e048c308e P 16440e365430405fb10366ab70e2b429: Initiating tablet copy from peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:13:49.187584 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:49.203012 7869 ts_tablet_manager.cc:902] T b7a2201f9605454ea12df4e3a7e5a63e P 8a666e798fe8477d8dd1e51b385d5ead: Initiating tablet copy from peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:13:49.260000 7868 tablet_copy_client.cc:308] T eb70e12d4fdb4d15a93d37dfdc659a14 P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.6:39975
I20240503 02:13:49.260298 7873 tablet_copy_client.cc:308] T 5ccde7688c4e49b5b9d89d6e048c308e P 16440e365430405fb10366ab70e2b429: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.7:44241
I20240503 02:13:49.260689 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:49.260872 457 tablet_copy_service.cc:140] P fbbc2be6382e47908ecb7c68c367bd26: Received BeginTabletCopySession request for tablet 5ccde7688c4e49b5b9d89d6e048c308e from peer 16440e365430405fb10366ab70e2b429 ({username='slave'} at 127.22.108.1:55517)
I20240503 02:13:49.261016 457 tablet_copy_service.cc:161] P fbbc2be6382e47908ecb7c68c367bd26: Beginning new tablet copy session on tablet 5ccde7688c4e49b5b9d89d6e048c308e from peer 16440e365430405fb10366ab70e2b429 at {username='slave'} at 127.22.108.1:55517: session id = 16440e365430405fb10366ab70e2b429-5ccde7688c4e49b5b9d89d6e048c308e
I20240503 02:13:49.261476 327 tablet_copy_service.cc:140] P 8a666e798fe8477d8dd1e51b385d5ead: Received BeginTabletCopySession request for tablet eb70e12d4fdb4d15a93d37dfdc659a14 from peer 155735c7f8b14997ac4c770aee2a9401 ({username='slave'} at 127.22.108.2:53863)
I20240503 02:13:49.261606 327 tablet_copy_service.cc:161] P 8a666e798fe8477d8dd1e51b385d5ead: Beginning new tablet copy session on tablet eb70e12d4fdb4d15a93d37dfdc659a14 from peer 155735c7f8b14997ac4c770aee2a9401 at {username='slave'} at 127.22.108.2:53863: session id = 155735c7f8b14997ac4c770aee2a9401-eb70e12d4fdb4d15a93d37dfdc659a14
I20240503 02:13:49.263990 457 tablet_copy_source_session.cc:205] T 5ccde7688c4e49b5b9d89d6e048c308e P fbbc2be6382e47908ecb7c68c367bd26: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:49.264199 327 tablet_copy_source_session.cc:205] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:49.265856 7868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb70e12d4fdb4d15a93d37dfdc659a14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:49.266861 7873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ccde7688c4e49b5b9d89d6e048c308e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:49.275743 7869 tablet_copy_client.cc:308] T b7a2201f9605454ea12df4e3a7e5a63e P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.2:37873
I20240503 02:13:49.276407 30026 tablet_copy_service.cc:140] P 155735c7f8b14997ac4c770aee2a9401: Received BeginTabletCopySession request for tablet b7a2201f9605454ea12df4e3a7e5a63e from peer 8a666e798fe8477d8dd1e51b385d5ead ({username='slave'} at 127.22.108.6:33785)
I20240503 02:13:49.276554 30026 tablet_copy_service.cc:161] P 155735c7f8b14997ac4c770aee2a9401: Beginning new tablet copy session on tablet b7a2201f9605454ea12df4e3a7e5a63e from peer 8a666e798fe8477d8dd1e51b385d5ead at {username='slave'} at 127.22.108.6:33785: session id = 8a666e798fe8477d8dd1e51b385d5ead-b7a2201f9605454ea12df4e3a7e5a63e
I20240503 02:13:49.279255 30026 tablet_copy_source_session.cc:205] T b7a2201f9605454ea12df4e3a7e5a63e P 155735c7f8b14997ac4c770aee2a9401: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:49.289582 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:49.289960 7869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7a2201f9605454ea12df4e3a7e5a63e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:49.294230 7873 tablet_copy_client.cc:783] T 5ccde7688c4e49b5b9d89d6e048c308e P 16440e365430405fb10366ab70e2b429: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:49.298657 7873 tablet_copy_client.cc:647] T 5ccde7688c4e49b5b9d89d6e048c308e P 16440e365430405fb10366ab70e2b429: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:49.301107 7868 tablet_copy_client.cc:783] T eb70e12d4fdb4d15a93d37dfdc659a14 P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:49.301450 7868 tablet_copy_client.cc:647] T eb70e12d4fdb4d15a93d37dfdc659a14 P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:49.310007 7869 tablet_copy_client.cc:783] T b7a2201f9605454ea12df4e3a7e5a63e P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:49.310257 7869 tablet_copy_client.cc:647] T b7a2201f9605454ea12df4e3a7e5a63e P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:49.313172 7869 tablet_copy_client.cc:523] T b7a2201f9605454ea12df4e3a7e5a63e P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:49.316748 7869 tablet_bootstrap.cc:492] T b7a2201f9605454ea12df4e3a7e5a63e P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:49.316915 7873 tablet_copy_client.cc:523] T 5ccde7688c4e49b5b9d89d6e048c308e P 16440e365430405fb10366ab70e2b429: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:49.318325 7868 tablet_copy_client.cc:523] T eb70e12d4fdb4d15a93d37dfdc659a14 P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:49.324218 7873 tablet_bootstrap.cc:492] T 5ccde7688c4e49b5b9d89d6e048c308e P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:13:49.325632 30266 raft_consensus.cc:1203] T 9c6e204df01e450598b05c3429574700 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:49.336973 7868 tablet_bootstrap.cc:492] T eb70e12d4fdb4d15a93d37dfdc659a14 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:13:49.353335 7873 tablet_bootstrap.cc:492] T 5ccde7688c4e49b5b9d89d6e048c308e P 16440e365430405fb10366ab70e2b429: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:49.353845 7873 tablet_bootstrap.cc:492] T 5ccde7688c4e49b5b9d89d6e048c308e P 16440e365430405fb10366ab70e2b429: Bootstrap complete.
I20240503 02:13:49.354133 7873 ts_tablet_manager.cc:1367] T 5ccde7688c4e49b5b9d89d6e048c308e P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.001s
I20240503 02:13:49.354821 7873 raft_consensus.cc:348] T 5ccde7688c4e49b5b9d89d6e048c308e P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:49.355127 7873 raft_consensus.cc:729] T 5ccde7688c4e49b5b9d89d6e048c308e P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:13:49.355355 7873 consensus_queue.cc:260] T 5ccde7688c4e49b5b9d89d6e048c308e P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:49.357216 7873 ts_tablet_manager.cc:1398] T 5ccde7688c4e49b5b9d89d6e048c308e P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:13:49.357954 457 tablet_copy_service.cc:342] P fbbc2be6382e47908ecb7c68c367bd26: Request end of tablet copy session 16440e365430405fb10366ab70e2b429-5ccde7688c4e49b5b9d89d6e048c308e received from {username='slave'} at 127.22.108.1:55517
I20240503 02:13:49.358062 457 tablet_copy_service.cc:433] P fbbc2be6382e47908ecb7c68c367bd26: ending tablet copy session 16440e365430405fb10366ab70e2b429-5ccde7688c4e49b5b9d89d6e048c308e on tablet 5ccde7688c4e49b5b9d89d6e048c308e with peer 16440e365430405fb10366ab70e2b429
I20240503 02:13:49.364401 7868 tablet_bootstrap.cc:492] T eb70e12d4fdb4d15a93d37dfdc659a14 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:49.365072 7868 tablet_bootstrap.cc:492] T eb70e12d4fdb4d15a93d37dfdc659a14 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap complete.
I20240503 02:13:49.365397 7868 ts_tablet_manager.cc:1367] T eb70e12d4fdb4d15a93d37dfdc659a14 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.005s
I20240503 02:13:49.366097 7868 raft_consensus.cc:348] T eb70e12d4fdb4d15a93d37dfdc659a14 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:49.366297 7868 raft_consensus.cc:729] T eb70e12d4fdb4d15a93d37dfdc659a14 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:13:49.366518 7868 consensus_queue.cc:260] T eb70e12d4fdb4d15a93d37dfdc659a14 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:49.367007 7868 ts_tablet_manager.cc:1398] T eb70e12d4fdb4d15a93d37dfdc659a14 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:13:49.367626 327 tablet_copy_service.cc:342] P 8a666e798fe8477d8dd1e51b385d5ead: Request end of tablet copy session 155735c7f8b14997ac4c770aee2a9401-eb70e12d4fdb4d15a93d37dfdc659a14 received from {username='slave'} at 127.22.108.2:53863
I20240503 02:13:49.367753 327 tablet_copy_service.cc:433] P 8a666e798fe8477d8dd1e51b385d5ead: ending tablet copy session 155735c7f8b14997ac4c770aee2a9401-eb70e12d4fdb4d15a93d37dfdc659a14 on tablet eb70e12d4fdb4d15a93d37dfdc659a14 with peer 155735c7f8b14997ac4c770aee2a9401
I20240503 02:13:49.378832 7869 tablet_bootstrap.cc:492] T b7a2201f9605454ea12df4e3a7e5a63e P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:49.379295 7869 tablet_bootstrap.cc:492] T b7a2201f9605454ea12df4e3a7e5a63e P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap complete.
I20240503 02:13:49.379562 7869 ts_tablet_manager.cc:1367] T b7a2201f9605454ea12df4e3a7e5a63e P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.063s user 0.039s sys 0.012s
I20240503 02:13:49.380302 7869 raft_consensus.cc:348] T b7a2201f9605454ea12df4e3a7e5a63e P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:49.380522 7869 raft_consensus.cc:729] T b7a2201f9605454ea12df4e3a7e5a63e P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:49.380759 7869 consensus_queue.cc:260] T b7a2201f9605454ea12df4e3a7e5a63e P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:49.381295 7869 ts_tablet_manager.cc:1398] T b7a2201f9605454ea12df4e3a7e5a63e P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:49.382074 30026 tablet_copy_service.cc:342] P 155735c7f8b14997ac4c770aee2a9401: Request end of tablet copy session 8a666e798fe8477d8dd1e51b385d5ead-b7a2201f9605454ea12df4e3a7e5a63e received from {username='slave'} at 127.22.108.6:33785
I20240503 02:13:49.382187 30026 tablet_copy_service.cc:433] P 155735c7f8b14997ac4c770aee2a9401: ending tablet copy session 8a666e798fe8477d8dd1e51b385d5ead-b7a2201f9605454ea12df4e3a7e5a63e on tablet b7a2201f9605454ea12df4e3a7e5a63e with peer 8a666e798fe8477d8dd1e51b385d5ead
I20240503 02:13:49.610141 30006 raft_consensus.cc:1203] T eb70e12d4fdb4d15a93d37dfdc659a14 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:49.626220 307 raft_consensus.cc:1203] T b7a2201f9605454ea12df4e3a7e5a63e P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:49.723356 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:49.735683 29873 raft_consensus.cc:1203] T 5ccde7688c4e49b5b9d89d6e048c308e P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:49.780325 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:49.780421 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:49.780680 30244 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:49.841826 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:49.854192 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:49.862776 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:49.954134 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:50.085338 30004 consensus_queue.cc:237] T b7a2201f9605454ea12df4e3a7e5a63e P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:50.087103 307 raft_consensus.cc:1261] T b7a2201f9605454ea12df4e3a7e5a63e P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:50.087150 29873 raft_consensus.cc:1261] T b7a2201f9605454ea12df4e3a7e5a63e P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:50.087602 7956 consensus_queue.cc:1035] T b7a2201f9605454ea12df4e3a7e5a63e P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:50.088160 7956 consensus_queue.cc:1035] T b7a2201f9605454ea12df4e3a7e5a63e P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:50.090588 7624 raft_consensus.cc:2932] T b7a2201f9605454ea12df4e3a7e5a63e P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:50.091233 307 raft_consensus.cc:2932] T b7a2201f9605454ea12df4e3a7e5a63e P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:50.093806 29873 raft_consensus.cc:2932] T b7a2201f9605454ea12df4e3a7e5a63e P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:50.096660 29727 catalog_manager.cc:5484] T b7a2201f9605454ea12df4e3a7e5a63e P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:50.116379 30114 tablet_service.cc:1514] Processing DeleteTablet for tablet b7a2201f9605454ea12df4e3a7e5a63e with delete_type TABLET_DATA_TOMBSTONED (TS a9094240860e43bfb129193f3bfd5be6 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44812
I20240503 02:13:50.117139 7991 tablet_replica.cc:330] T b7a2201f9605454ea12df4e3a7e5a63e P a9094240860e43bfb129193f3bfd5be6: stopping tablet replica
I20240503 02:13:50.117471 7991 raft_consensus.cc:2229] T b7a2201f9605454ea12df4e3a7e5a63e P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:50.117785 7991 raft_consensus.cc:2258] T b7a2201f9605454ea12df4e3a7e5a63e P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:50.118706 7991 ts_tablet_manager.cc:1875] T b7a2201f9605454ea12df4e3a7e5a63e P a9094240860e43bfb129193f3bfd5be6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:50.125571 7991 ts_tablet_manager.cc:1888] T b7a2201f9605454ea12df4e3a7e5a63e P a9094240860e43bfb129193f3bfd5be6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:13:50.125685 7991 log.cc:1196] T b7a2201f9605454ea12df4e3a7e5a63e P a9094240860e43bfb129193f3bfd5be6: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/wals/b7a2201f9605454ea12df4e3a7e5a63e
I20240503 02:13:50.126435 29718 catalog_manager.cc:4830] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): tablet b7a2201f9605454ea12df4e3a7e5a63e (table rebalance_test_table_5 [id=99dbec2a2b63467ab34b96ecc8479244]) successfully deleted
I20240503 02:13:50.312515 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:50.354271 30244 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:50.358129 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:50.371273 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:50.414690 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:50.432047 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:50.446880 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:50.452853 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:50.629452 434 consensus_queue.cc:237] T 9c6e204df01e450598b05c3429574700 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:50.631637 306 raft_consensus.cc:1261] T 9c6e204df01e450598b05c3429574700 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:50.631729 30266 raft_consensus.cc:1261] T 9c6e204df01e450598b05c3429574700 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:50.632514 7952 consensus_queue.cc:1035] T 9c6e204df01e450598b05c3429574700 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:50.633067 6997 consensus_queue.cc:1035] T 9c6e204df01e450598b05c3429574700 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:50.635537 7952 raft_consensus.cc:2932] T 9c6e204df01e450598b05c3429574700 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:50.636247 30266 raft_consensus.cc:2932] T 9c6e204df01e450598b05c3429574700 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:50.636874 306 raft_consensus.cc:2932] T 9c6e204df01e450598b05c3429574700 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:50.643330 29734 catalog_manager.cc:5484] T 9c6e204df01e450598b05c3429574700 P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:50.657181 547 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c6e204df01e450598b05c3429574700 with delete_type TABLET_DATA_TOMBSTONED (TS f575ce92c0224eeb80b9fae5f6561fbe not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:39028
I20240503 02:13:50.657792 8071 tablet_replica.cc:330] T 9c6e204df01e450598b05c3429574700 P f575ce92c0224eeb80b9fae5f6561fbe: stopping tablet replica
I20240503 02:13:50.658053 8071 raft_consensus.cc:2229] T 9c6e204df01e450598b05c3429574700 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:50.658277 8071 raft_consensus.cc:2258] T 9c6e204df01e450598b05c3429574700 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:50.658836 8071 ts_tablet_manager.cc:1875] T 9c6e204df01e450598b05c3429574700 P f575ce92c0224eeb80b9fae5f6561fbe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:50.665196 8071 ts_tablet_manager.cc:1888] T 9c6e204df01e450598b05c3429574700 P f575ce92c0224eeb80b9fae5f6561fbe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:13:50.665318 8071 log.cc:1196] T 9c6e204df01e450598b05c3429574700 P f575ce92c0224eeb80b9fae5f6561fbe: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal/wals/9c6e204df01e450598b05c3429574700
I20240503 02:13:50.665961 29718 catalog_manager.cc:4830] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): tablet 9c6e204df01e450598b05c3429574700 (table rebalancer_extra_table_20 [id=9fae481dc13a491db2520bd2d31be634]) successfully deleted
I20240503 02:13:50.926479 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:50.958386 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:50.982657 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:50.985291 30244 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:51.109010 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:51.136790 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 02:13:51.140064 29809 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.5:35595, user_credentials={real_user=slave}} blocked reactor thread for 56110.2us
I20240503 02:13:51.181461 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:51.192996 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:51.309247 29873 consensus_queue.cc:237] T c97c7a843366426887115e13f35faf52 P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:13:51.311245 434 raft_consensus.cc:1261] T c97c7a843366426887115e13f35faf52 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:51.311373 306 raft_consensus.cc:1261] T c97c7a843366426887115e13f35faf52 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:51.311861 8127 consensus_queue.cc:1035] T c97c7a843366426887115e13f35faf52 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:51.312165 7576 consensus_queue.cc:1035] T c97c7a843366426887115e13f35faf52 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:51.315138 8127 raft_consensus.cc:2932] T c97c7a843366426887115e13f35faf52 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:51.317622 434 raft_consensus.cc:2932] T c97c7a843366426887115e13f35faf52 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:51.323350 29734 catalog_manager.cc:5484] T c97c7a843366426887115e13f35faf52 P 16440e365430405fb10366ab70e2b429 reported cstate change: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:51.326659 306 raft_consensus.cc:2932] T c97c7a843366426887115e13f35faf52 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:13:51.345705 32619 tablet_service.cc:1514] Processing DeleteTablet for tablet c97c7a843366426887115e13f35faf52 with delete_type TABLET_DATA_TOMBSTONED (TS 9a4d92bc25374fb3960fde7d10be309d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:38200
I20240503 02:13:51.349485 8162 tablet_replica.cc:330] T c97c7a843366426887115e13f35faf52 P 9a4d92bc25374fb3960fde7d10be309d: stopping tablet replica
I20240503 02:13:51.351773 8162 raft_consensus.cc:2229] T c97c7a843366426887115e13f35faf52 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:51.351998 8162 raft_consensus.cc:2258] T c97c7a843366426887115e13f35faf52 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:51.352638 8162 ts_tablet_manager.cc:1875] T c97c7a843366426887115e13f35faf52 P 9a4d92bc25374fb3960fde7d10be309d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:51.363078 8162 ts_tablet_manager.cc:1888] T c97c7a843366426887115e13f35faf52 P 9a4d92bc25374fb3960fde7d10be309d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:13:51.363313 8162 log.cc:1196] T c97c7a843366426887115e13f35faf52 P 9a4d92bc25374fb3960fde7d10be309d: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal/wals/c97c7a843366426887115e13f35faf52
I20240503 02:13:51.364979 29719 catalog_manager.cc:4830] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): tablet c97c7a843366426887115e13f35faf52 (table rebalancer_extra_table_18 [id=3e0093624a2d4e6fa34244d26a961331]) successfully deleted
I20240503 02:13:51.678968 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:51.699275 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:51.708500 30244 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:51.729764 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:51.751758 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:51.760643 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:51.786999 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:51.812516 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:52.041638 433 consensus_queue.cc:237] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:52.044595 567 raft_consensus.cc:1261] T 48b76d7755294d37860842a4a73ab28b P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:52.044740 30004 raft_consensus.cc:1261] T 48b76d7755294d37860842a4a73ab28b P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:52.045365 6997 consensus_queue.cc:1035] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:52.045841 7952 consensus_queue.cc:1035] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:52.049710 6997 raft_consensus.cc:2932] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:52.050271 30004 raft_consensus.cc:2932] T 48b76d7755294d37860842a4a73ab28b P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:52.056381 29729 catalog_manager.cc:5484] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: config changed from index 2 to 3, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:52.057502 567 raft_consensus.cc:2932] T 48b76d7755294d37860842a4a73ab28b P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:52.067885 30114 tablet_service.cc:1514] Processing DeleteTablet for tablet 48b76d7755294d37860842a4a73ab28b with delete_type TABLET_DATA_TOMBSTONED (TS a9094240860e43bfb129193f3bfd5be6 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44812
I20240503 02:13:52.068910 8240 tablet_replica.cc:330] T 48b76d7755294d37860842a4a73ab28b P a9094240860e43bfb129193f3bfd5be6: stopping tablet replica
I20240503 02:13:52.069171 8240 raft_consensus.cc:2229] T 48b76d7755294d37860842a4a73ab28b P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:52.069438 8240 raft_consensus.cc:2258] T 48b76d7755294d37860842a4a73ab28b P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:52.070037 8240 ts_tablet_manager.cc:1875] T 48b76d7755294d37860842a4a73ab28b P a9094240860e43bfb129193f3bfd5be6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:52.082105 8240 ts_tablet_manager.cc:1888] T 48b76d7755294d37860842a4a73ab28b P a9094240860e43bfb129193f3bfd5be6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:13:52.082262 8240 log.cc:1196] T 48b76d7755294d37860842a4a73ab28b P a9094240860e43bfb129193f3bfd5be6: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/wals/48b76d7755294d37860842a4a73ab28b
I20240503 02:13:52.083026 29718 catalog_manager.cc:4830] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): tablet 48b76d7755294d37860842a4a73ab28b (table rebalancer_extra_table_48 [id=196b7559fe174e1b963b8d38f2240e48]) successfully deleted
I20240503 02:13:52.351166 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:52.372406 30244 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:52.372993 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:52.426738 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:52.443898 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:52.511184 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:52.548226 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:52.609306 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:52.693775 30133 consensus_queue.cc:237] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:52.695744 30005 raft_consensus.cc:1261] T c5cd64614c53487ea1c6cc4a4ca5c379 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:52.695735 32640 raft_consensus.cc:1261] T c5cd64614c53487ea1c6cc4a4ca5c379 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:52.696209 7599 consensus_queue.cc:1035] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:52.696465 8312 consensus_queue.cc:1035] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:52.706265 7599 raft_consensus.cc:2932] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:52.707059 30005 raft_consensus.cc:2932] T c5cd64614c53487ea1c6cc4a4ca5c379 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:52.713866 29729 catalog_manager.cc:5484] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 reported cstate change: config changed from index 4 to 5, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:52.715893 32640 raft_consensus.cc:2932] T c5cd64614c53487ea1c6cc4a4ca5c379 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:52.726856 29853 tablet_service.cc:1514] Processing DeleteTablet for tablet c5cd64614c53487ea1c6cc4a4ca5c379 with delete_type TABLET_DATA_TOMBSTONED (TS 16440e365430405fb10366ab70e2b429 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:58744
I20240503 02:13:52.728303 8323 tablet_replica.cc:330] T c5cd64614c53487ea1c6cc4a4ca5c379 P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:13:52.728544 8323 raft_consensus.cc:2229] T c5cd64614c53487ea1c6cc4a4ca5c379 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:52.729310 8323 raft_consensus.cc:2258] T c5cd64614c53487ea1c6cc4a4ca5c379 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:52.730023 8323 ts_tablet_manager.cc:1875] T c5cd64614c53487ea1c6cc4a4ca5c379 P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:52.735239 8323 ts_tablet_manager.cc:1888] T c5cd64614c53487ea1c6cc4a4ca5c379 P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:13:52.735342 8323 log.cc:1196] T c5cd64614c53487ea1c6cc4a4ca5c379 P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/c5cd64614c53487ea1c6cc4a4ca5c379
I20240503 02:13:52.736114 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet c5cd64614c53487ea1c6cc4a4ca5c379 (table rebalance_test_table_19 [id=3c36e49b52c04307a655de366327db8c]) successfully deleted
I20240503 02:13:53.079932 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:53.084074 30244 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:53.084504 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:53.089597 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:53.100657 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:53.117484 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:53.212873 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:53.261049 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:53.353070 307 consensus_queue.cc:237] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:53.356680 30005 raft_consensus.cc:1261] T ac4d497de05c49edbd6e0edac73c058d P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:53.356829 565 raft_consensus.cc:1261] T ac4d497de05c49edbd6e0edac73c058d P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:53.357520 7476 consensus_queue.cc:1035] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:53.358011 8277 consensus_queue.cc:1035] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:53.360464 7476 raft_consensus.cc:2932] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:53.362982 565 raft_consensus.cc:2932] T ac4d497de05c49edbd6e0edac73c058d P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:53.380824 29732 catalog_manager.cc:5484] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:53.395798 30004 raft_consensus.cc:2932] T ac4d497de05c49edbd6e0edac73c058d P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:53.405808 415 tablet_service.cc:1514] Processing DeleteTablet for tablet ac4d497de05c49edbd6e0edac73c058d with delete_type TABLET_DATA_TOMBSTONED (TS fbbc2be6382e47908ecb7c68c367bd26 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50814
I20240503 02:13:53.425597 8425 tablet_replica.cc:330] T ac4d497de05c49edbd6e0edac73c058d P fbbc2be6382e47908ecb7c68c367bd26: stopping tablet replica
I20240503 02:13:53.425841 8425 raft_consensus.cc:2229] T ac4d497de05c49edbd6e0edac73c058d P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:53.426031 8425 raft_consensus.cc:2258] T ac4d497de05c49edbd6e0edac73c058d P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:53.426653 8425 ts_tablet_manager.cc:1875] T ac4d497de05c49edbd6e0edac73c058d P fbbc2be6382e47908ecb7c68c367bd26: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:53.440325 8425 ts_tablet_manager.cc:1888] T ac4d497de05c49edbd6e0edac73c058d P fbbc2be6382e47908ecb7c68c367bd26: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:13:53.440454 8425 log.cc:1196] T ac4d497de05c49edbd6e0edac73c058d P fbbc2be6382e47908ecb7c68c367bd26: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal/wals/ac4d497de05c49edbd6e0edac73c058d
I20240503 02:13:53.442001 29719 catalog_manager.cc:4830] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): tablet ac4d497de05c49edbd6e0edac73c058d (table rebalancer_extra_table_28 [id=d70cb2e6817a47df8925321a6a778740]) successfully deleted
I20240503 02:13:53.638935 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:53.655176 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:53.700721 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:53.701223 30244 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:53.712952 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:53.725256 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:53.765949 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:53.771288 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 02:13:53.788817 30072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.1:36719, user_credentials={real_user=slave}} blocked reactor thread for 48213.6us
I20240503 02:13:53.936823 30264 consensus_queue.cc:237] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:53.939762 29873 raft_consensus.cc:1261] T b76b807d2eb24f3ead50bdc0bb1b0231 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:53.939834 30132 raft_consensus.cc:1261] T b76b807d2eb24f3ead50bdc0bb1b0231 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:53.940308 7241 consensus_queue.cc:1035] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:53.940573 8209 consensus_queue.cc:1035] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:53.950659 7241 raft_consensus.cc:2932] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:53.953691 30133 raft_consensus.cc:2932] T b76b807d2eb24f3ead50bdc0bb1b0231 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:53.959640 29873 raft_consensus.cc:2932] T b76b807d2eb24f3ead50bdc0bb1b0231 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:53.961330 29727 catalog_manager.cc:5484] T b76b807d2eb24f3ead50bdc0bb1b0231 P a9094240860e43bfb129193f3bfd5be6 reported cstate change: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:53.986016 547 tablet_service.cc:1514] Processing DeleteTablet for tablet b76b807d2eb24f3ead50bdc0bb1b0231 with delete_type TABLET_DATA_TOMBSTONED (TS f575ce92c0224eeb80b9fae5f6561fbe not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:39028
I20240503 02:13:53.989784 8503 tablet_replica.cc:330] T b76b807d2eb24f3ead50bdc0bb1b0231 P f575ce92c0224eeb80b9fae5f6561fbe: stopping tablet replica
I20240503 02:13:53.990096 8503 raft_consensus.cc:2229] T b76b807d2eb24f3ead50bdc0bb1b0231 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:53.990356 8503 raft_consensus.cc:2258] T b76b807d2eb24f3ead50bdc0bb1b0231 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:53.991210 8503 ts_tablet_manager.cc:1875] T b76b807d2eb24f3ead50bdc0bb1b0231 P f575ce92c0224eeb80b9fae5f6561fbe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:53.998939 8503 ts_tablet_manager.cc:1888] T b76b807d2eb24f3ead50bdc0bb1b0231 P f575ce92c0224eeb80b9fae5f6561fbe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:13:53.999130 8503 log.cc:1196] T b76b807d2eb24f3ead50bdc0bb1b0231 P f575ce92c0224eeb80b9fae5f6561fbe: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal/wals/b76b807d2eb24f3ead50bdc0bb1b0231
I20240503 02:13:53.999859 29718 catalog_manager.cc:4830] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): tablet b76b807d2eb24f3ead50bdc0bb1b0231 (table rebalancer_extra_table_30 [id=854d24d23fe34d699e6656ef51d15693]) successfully deleted
I20240503 02:13:54.231086 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:54.240882 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:54.254426 30244 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:54.273283 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:54.283056 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:54.283165 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:54.322558 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:54.358472 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:54.549660 434 consensus_queue.cc:237] T 4cc7861a21f94dabb01133bac9ba0b46 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:54.551695 30004 raft_consensus.cc:1261] T 4cc7861a21f94dabb01133bac9ba0b46 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:54.551705 30265 raft_consensus.cc:1261] T 4cc7861a21f94dabb01133bac9ba0b46 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:54.552377 8391 consensus_queue.cc:1035] T 4cc7861a21f94dabb01133bac9ba0b46 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:54.552680 8232 consensus_queue.cc:1035] T 4cc7861a21f94dabb01133bac9ba0b46 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:54.556375 8391 raft_consensus.cc:2932] T 4cc7861a21f94dabb01133bac9ba0b46 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:54.557561 30004 raft_consensus.cc:2932] T 4cc7861a21f94dabb01133bac9ba0b46 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:54.557693 30265 raft_consensus.cc:2932] T 4cc7861a21f94dabb01133bac9ba0b46 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:54.564918 29729 catalog_manager.cc:5484] T 4cc7861a21f94dabb01133bac9ba0b46 P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:54.571498 32619 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cc7861a21f94dabb01133bac9ba0b46 with delete_type TABLET_DATA_TOMBSTONED (TS 9a4d92bc25374fb3960fde7d10be309d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:38200
I20240503 02:13:54.573243 8587 tablet_replica.cc:330] T 4cc7861a21f94dabb01133bac9ba0b46 P 9a4d92bc25374fb3960fde7d10be309d: stopping tablet replica
I20240503 02:13:54.573487 8587 raft_consensus.cc:2229] T 4cc7861a21f94dabb01133bac9ba0b46 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:54.573709 8587 raft_consensus.cc:2258] T 4cc7861a21f94dabb01133bac9ba0b46 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:54.574362 8587 ts_tablet_manager.cc:1875] T 4cc7861a21f94dabb01133bac9ba0b46 P 9a4d92bc25374fb3960fde7d10be309d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:54.580426 8587 ts_tablet_manager.cc:1888] T 4cc7861a21f94dabb01133bac9ba0b46 P 9a4d92bc25374fb3960fde7d10be309d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:13:54.580588 8587 log.cc:1196] T 4cc7861a21f94dabb01133bac9ba0b46 P 9a4d92bc25374fb3960fde7d10be309d: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal/wals/4cc7861a21f94dabb01133bac9ba0b46
I20240503 02:13:54.581606 29719 catalog_manager.cc:4830] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): tablet 4cc7861a21f94dabb01133bac9ba0b46 (table rebalancer_extra_table_4 [id=002e01fddb8349d99a20cae11d727d7d]) successfully deleted
I20240503 02:13:54.802628 30244 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:54.814966 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:54.819185 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:54.827589 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:54.853149 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:54.854995 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:54.888109 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:54.910737 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:55.084901 567 consensus_queue.cc:237] T c2d33db64ee8482aad840af219d60b8b P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:13:55.087074 29873 raft_consensus.cc:1261] T c2d33db64ee8482aad840af219d60b8b P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:55.087118 30132 raft_consensus.cc:1261] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:55.087642 8478 consensus_queue.cc:1035] T c2d33db64ee8482aad840af219d60b8b P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:55.087929 8344 consensus_queue.cc:1035] T c2d33db64ee8482aad840af219d60b8b P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:55.091526 8478 raft_consensus.cc:2932] T c2d33db64ee8482aad840af219d60b8b P f575ce92c0224eeb80b9fae5f6561fbe [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:55.093469 30132 raft_consensus.cc:2932] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:55.093473 29873 raft_consensus.cc:2932] T c2d33db64ee8482aad840af219d60b8b P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:55.097946 29728 catalog_manager.cc:5484] T c2d33db64ee8482aad840af219d60b8b P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New cstate: current_term: 1 leader_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:13:55.109372 415 tablet_service.cc:1514] Processing DeleteTablet for tablet c2d33db64ee8482aad840af219d60b8b with delete_type TABLET_DATA_TOMBSTONED (TS fbbc2be6382e47908ecb7c68c367bd26 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50814
I20240503 02:13:55.109999 8672 tablet_replica.cc:330] T c2d33db64ee8482aad840af219d60b8b P fbbc2be6382e47908ecb7c68c367bd26: stopping tablet replica
I20240503 02:13:55.110216 8672 raft_consensus.cc:2229] T c2d33db64ee8482aad840af219d60b8b P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:55.110455 8672 raft_consensus.cc:2258] T c2d33db64ee8482aad840af219d60b8b P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:55.111158 8672 ts_tablet_manager.cc:1875] T c2d33db64ee8482aad840af219d60b8b P fbbc2be6382e47908ecb7c68c367bd26: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:55.117741 8672 ts_tablet_manager.cc:1888] T c2d33db64ee8482aad840af219d60b8b P fbbc2be6382e47908ecb7c68c367bd26: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:13:55.117960 8672 log.cc:1196] T c2d33db64ee8482aad840af219d60b8b P fbbc2be6382e47908ecb7c68c367bd26: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal/wals/c2d33db64ee8482aad840af219d60b8b
I20240503 02:13:55.118650 29719 catalog_manager.cc:4830] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): tablet c2d33db64ee8482aad840af219d60b8b (table rebalancer_extra_table_4 [id=002e01fddb8349d99a20cae11d727d7d]) successfully deleted
I20240503 02:13:55.352442 29985 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:55.369184 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:55.418915 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240503 02:13:55.478463 32573 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.3:42679, user_credentials={real_user=slave}} blocked reactor thread for 85815.5us
I20240503 02:13:55.481647 30244 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:55.509471 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240503 02:13:55.522361 30199 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.3:42679, user_credentials={real_user=slave}} blocked reactor thread for 131945us
I20240503 02:13:55.543136 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:55.575747 30114 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:55.586468 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:55.741533 30004 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "db0756f0f331402aaef0583e1f9e5318"
dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47320
I20240503 02:13:55.741770 30004 raft_consensus.cc:595] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:13:55.887742 8698 raft_consensus.cc:979] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401: : Instructing follower a9094240860e43bfb129193f3bfd5be6 to start an election
I20240503 02:13:55.887921 8698 raft_consensus.cc:1067] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Signalling peer a9094240860e43bfb129193f3bfd5be6 to start an election
I20240503 02:13:55.888995 30132 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "db0756f0f331402aaef0583e1f9e5318"
dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
from {username='slave'} at 127.22.108.2:56649
I20240503 02:13:55.889267 30132 raft_consensus.cc:482] T db0756f0f331402aaef0583e1f9e5318 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:55.889367 30132 raft_consensus.cc:3037] T db0756f0f331402aaef0583e1f9e5318 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:55.892091 30132 raft_consensus.cc:504] T db0756f0f331402aaef0583e1f9e5318 P a9094240860e43bfb129193f3bfd5be6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:55.896310 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db0756f0f331402aaef0583e1f9e5318" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:13:55.896523 30004 raft_consensus.cc:3032] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:55.896612 30004 raft_consensus.cc:729] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Running, Role: LEADER
I20240503 02:13:55.896996 30004 consensus_queue.cc:260] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:55.897186 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db0756f0f331402aaef0583e1f9e5318" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:13:55.897378 30264 raft_consensus.cc:3037] T db0756f0f331402aaef0583e1f9e5318 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:55.897550 30004 raft_consensus.cc:3037] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:55.900092 30264 raft_consensus.cc:2454] T db0756f0f331402aaef0583e1f9e5318 P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 2.
I20240503 02:13:55.903667 30004 raft_consensus.cc:2454] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 2.
I20240503 02:13:55.904147 30070 leader_election.cc:302] T db0756f0f331402aaef0583e1f9e5318 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 1f10f80f13c84e80b03eda7085a3b012, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:13:55.904495 8477 raft_consensus.cc:2781] T db0756f0f331402aaef0583e1f9e5318 P a9094240860e43bfb129193f3bfd5be6 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:55.904611 8477 raft_consensus.cc:686] T db0756f0f331402aaef0583e1f9e5318 P a9094240860e43bfb129193f3bfd5be6 [term 2 LEADER]: Becoming Leader. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Running, Role: LEADER
I20240503 02:13:55.904877 8477 consensus_queue.cc:237] T db0756f0f331402aaef0583e1f9e5318 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:55.910671 30132 leader_election.cc:288] T db0756f0f331402aaef0583e1f9e5318 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 election: Requested vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:55.911679 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db0756f0f331402aaef0583e1f9e5318" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:13:55.911881 307 raft_consensus.cc:3037] T db0756f0f331402aaef0583e1f9e5318 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:55.914608 307 raft_consensus.cc:2454] T db0756f0f331402aaef0583e1f9e5318 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 2.
I20240503 02:13:55.936790 29728 catalog_manager.cc:5484] T db0756f0f331402aaef0583e1f9e5318 P a9094240860e43bfb129193f3bfd5be6 reported cstate change: term changed from 1 to 2, leader changed from 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) to a9094240860e43bfb129193f3bfd5be6 (127.22.108.3). New cstate: current_term: 2 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:56.065655 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:13:56.082736 30244 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:56.094156 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:56.099301 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:56.109828 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:56.132540 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:56.140146 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:56.172375 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:56.318300 307 raft_consensus.cc:1261] T db0756f0f331402aaef0583e1f9e5318 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:13:56.318929 8312 consensus_queue.cc:1035] T db0756f0f331402aaef0583e1f9e5318 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:56.482543 30004 raft_consensus.cc:1261] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401 [term 2 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:13:56.483126 8312 consensus_queue.cc:1035] T db0756f0f331402aaef0583e1f9e5318 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:56.507761 30264 raft_consensus.cc:1261] T db0756f0f331402aaef0583e1f9e5318 P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:13:56.508492 8477 consensus_queue.cc:1035] T db0756f0f331402aaef0583e1f9e5318 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 02:13:56.526407 502 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.6:39975, user_credentials={real_user=slave}} blocked reactor thread for 50225us
I20240503 02:13:56.738173 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:56.757611 30244 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:56.766974 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:13:56.791872 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:56.806542 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:56.833189 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:56.839715 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:56.858213 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:57.062966 29873 consensus_queue.cc:237] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:13:57.065064 565 raft_consensus.cc:1261] T 46d608d84863424e8c65feac7c2e835f P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:57.065061 30004 raft_consensus.cc:1261] T 46d608d84863424e8c65feac7c2e835f P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:57.065573 8644 consensus_queue.cc:1035] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:57.065827 8770 consensus_queue.cc:1035] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:57.068650 8644 raft_consensus.cc:2932] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:57.069461 565 raft_consensus.cc:2932] T 46d608d84863424e8c65feac7c2e835f P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:57.070245 30004 raft_consensus.cc:2932] T 46d608d84863424e8c65feac7c2e835f P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:57.075863 29729 catalog_manager.cc:5484] T 46d608d84863424e8c65feac7c2e835f P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:13:57.087144 30114 tablet_service.cc:1514] Processing DeleteTablet for tablet 46d608d84863424e8c65feac7c2e835f with delete_type TABLET_DATA_TOMBSTONED (TS a9094240860e43bfb129193f3bfd5be6 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44812
I20240503 02:13:57.087963 8910 tablet_replica.cc:330] T 46d608d84863424e8c65feac7c2e835f P a9094240860e43bfb129193f3bfd5be6: stopping tablet replica
I20240503 02:13:57.088254 8910 raft_consensus.cc:2229] T 46d608d84863424e8c65feac7c2e835f P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:57.090555 8910 raft_consensus.cc:2258] T 46d608d84863424e8c65feac7c2e835f P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:57.091395 8910 ts_tablet_manager.cc:1875] T 46d608d84863424e8c65feac7c2e835f P a9094240860e43bfb129193f3bfd5be6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:57.097196 8910 ts_tablet_manager.cc:1888] T 46d608d84863424e8c65feac7c2e835f P a9094240860e43bfb129193f3bfd5be6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:13:57.097326 8910 log.cc:1196] T 46d608d84863424e8c65feac7c2e835f P a9094240860e43bfb129193f3bfd5be6: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/wals/46d608d84863424e8c65feac7c2e835f
I20240503 02:13:57.098168 29718 catalog_manager.cc:4830] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): tablet 46d608d84863424e8c65feac7c2e835f (table rebalance_test_table_1 [id=28d7eeaed8114076b7afb44e4f89a2ea]) successfully deleted
I20240503 02:13:57.360733 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:13:57.376431 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:57.388926 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:57.405737 30244 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:57.449522 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:57.452159 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:57.523474 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:57.532003 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:57.690044 307 consensus_queue.cc:237] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:57.692370 30004 raft_consensus.cc:1261] T eb70e12d4fdb4d15a93d37dfdc659a14 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:57.692476 567 raft_consensus.cc:1261] T eb70e12d4fdb4d15a93d37dfdc659a14 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:57.692901 8798 consensus_queue.cc:1035] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:57.693153 8277 consensus_queue.cc:1035] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:57.695601 8277 raft_consensus.cc:2932] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:57.696102 567 raft_consensus.cc:2932] T eb70e12d4fdb4d15a93d37dfdc659a14 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:57.696501 30004 raft_consensus.cc:2932] T eb70e12d4fdb4d15a93d37dfdc659a14 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:57.703089 29730 catalog_manager.cc:5484] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:57.710211 415 tablet_service.cc:1514] Processing DeleteTablet for tablet eb70e12d4fdb4d15a93d37dfdc659a14 with delete_type TABLET_DATA_TOMBSTONED (TS fbbc2be6382e47908ecb7c68c367bd26 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50814
I20240503 02:13:57.713094 9007 tablet_replica.cc:330] T eb70e12d4fdb4d15a93d37dfdc659a14 P fbbc2be6382e47908ecb7c68c367bd26: stopping tablet replica
I20240503 02:13:57.713375 9007 raft_consensus.cc:2229] T eb70e12d4fdb4d15a93d37dfdc659a14 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:57.713712 9007 raft_consensus.cc:2258] T eb70e12d4fdb4d15a93d37dfdc659a14 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:57.715150 9007 ts_tablet_manager.cc:1875] T eb70e12d4fdb4d15a93d37dfdc659a14 P fbbc2be6382e47908ecb7c68c367bd26: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:57.720530 9007 ts_tablet_manager.cc:1888] T eb70e12d4fdb4d15a93d37dfdc659a14 P fbbc2be6382e47908ecb7c68c367bd26: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:13:57.720646 9007 log.cc:1196] T eb70e12d4fdb4d15a93d37dfdc659a14 P fbbc2be6382e47908ecb7c68c367bd26: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal/wals/eb70e12d4fdb4d15a93d37dfdc659a14
I20240503 02:13:57.721372 29719 catalog_manager.cc:4830] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): tablet eb70e12d4fdb4d15a93d37dfdc659a14 (table rebalancer_extra_table_24 [id=782c508495e74448b6859bfc4ff35de0]) successfully deleted
I20240503 02:13:58.109534 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:13:58.141959 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:58.151881 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:58.199347 30244 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:58.224745 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:58.243119 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:58.253340 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:58.301422 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:58.429001 30004 consensus_queue.cc:237] T 51f5f7297e6448d0b0e9afeedad01429 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:13:58.431880 307 raft_consensus.cc:1261] T 51f5f7297e6448d0b0e9afeedad01429 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:58.432185 30264 raft_consensus.cc:1261] T 51f5f7297e6448d0b0e9afeedad01429 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:58.432421 9050 consensus_queue.cc:1035] T 51f5f7297e6448d0b0e9afeedad01429 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:58.432657 8698 consensus_queue.cc:1035] T 51f5f7297e6448d0b0e9afeedad01429 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:58.438316 9099 raft_consensus.cc:2932] T 51f5f7297e6448d0b0e9afeedad01429 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:58.440469 307 raft_consensus.cc:2932] T 51f5f7297e6448d0b0e9afeedad01429 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:58.444061 30265 raft_consensus.cc:2932] T 51f5f7297e6448d0b0e9afeedad01429 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:58.446164 29727 catalog_manager.cc:5484] T 51f5f7297e6448d0b0e9afeedad01429 P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:13:58.461149 415 tablet_service.cc:1514] Processing DeleteTablet for tablet 51f5f7297e6448d0b0e9afeedad01429 with delete_type TABLET_DATA_TOMBSTONED (TS fbbc2be6382e47908ecb7c68c367bd26 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50814
I20240503 02:13:58.462510 9108 tablet_replica.cc:330] T 51f5f7297e6448d0b0e9afeedad01429 P fbbc2be6382e47908ecb7c68c367bd26: stopping tablet replica
I20240503 02:13:58.462862 9108 raft_consensus.cc:2229] T 51f5f7297e6448d0b0e9afeedad01429 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:58.463160 9108 raft_consensus.cc:2258] T 51f5f7297e6448d0b0e9afeedad01429 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:58.463933 9108 ts_tablet_manager.cc:1875] T 51f5f7297e6448d0b0e9afeedad01429 P fbbc2be6382e47908ecb7c68c367bd26: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:58.472123 9108 ts_tablet_manager.cc:1888] T 51f5f7297e6448d0b0e9afeedad01429 P fbbc2be6382e47908ecb7c68c367bd26: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:13:58.472265 9108 log.cc:1196] T 51f5f7297e6448d0b0e9afeedad01429 P fbbc2be6382e47908ecb7c68c367bd26: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal/wals/51f5f7297e6448d0b0e9afeedad01429
I20240503 02:13:58.473263 29719 catalog_manager.cc:4830] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): tablet 51f5f7297e6448d0b0e9afeedad01429 (table rebalancer_extra_table_22 [id=fedaa5e192b44061a6bd08563b273009]) successfully deleted
I20240503 02:13:58.513309 306 consensus_queue.cc:237] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:58.516372 30265 raft_consensus.cc:1261] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:58.516372 565 raft_consensus.cc:1261] T 5a44b34b585b47528c22fc678289b01a P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:13:58.517236 32711 consensus_peers.cc:485] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Couldn't send request to peer 16440e365430405fb10366ab70e2b429. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5a44b34b585b47528c22fc678289b01a. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:58.517454 9128 consensus_queue.cc:1035] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:58.518182 9128 consensus_queue.cc:1035] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:58.522353 8798 raft_consensus.cc:2932] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:58.522998 567 raft_consensus.cc:2932] T 5a44b34b585b47528c22fc678289b01a P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:58.530344 29725 catalog_manager.cc:5484] T 5a44b34b585b47528c22fc678289b01a P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:58.530640 30265 raft_consensus.cc:2932] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:58.565757 306 consensus_queue.cc:237] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:58.567842 30004 raft_consensus.cc:1261] T 480f7b57c2c04c1399b945a023fbe41f P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:58.568194 30265 raft_consensus.cc:1261] T 480f7b57c2c04c1399b945a023fbe41f P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:13:58.568486 32709 consensus_peers.cc:485] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 480f7b57c2c04c1399b945a023fbe41f. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:58.568746 8798 consensus_queue.cc:1035] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:58.568992 9127 consensus_queue.cc:1035] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:58.572482 8798 raft_consensus.cc:2932] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:58.572999 30265 raft_consensus.cc:2932] T 480f7b57c2c04c1399b945a023fbe41f P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:58.577945 30005 raft_consensus.cc:2932] T 480f7b57c2c04c1399b945a023fbe41f P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:58.579480 29725 catalog_manager.cc:5484] T 480f7b57c2c04c1399b945a023fbe41f P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: config changed from index -1 to 2, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:13:58.619670 32641 consensus_queue.cc:237] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:58.622088 567 raft_consensus.cc:1261] T fb8fcfa3f3ab46b182b692a0c4fc61ed P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:58.622797 30264 raft_consensus.cc:1261] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:13:58.623613 32574 consensus_peers.cc:485] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 9a4d92bc25374fb3960fde7d10be309d -> Peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): Couldn't send request to peer 8a666e798fe8477d8dd1e51b385d5ead. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb8fcfa3f3ab46b182b692a0c4fc61ed. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:58.623885 8926 consensus_queue.cc:1035] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:58.624210 8726 consensus_queue.cc:1035] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:58.629410 8926 raft_consensus.cc:2932] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 9a4d92bc25374fb3960fde7d10be309d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:58.631979 30264 raft_consensus.cc:2932] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:58.637209 567 raft_consensus.cc:2932] T fb8fcfa3f3ab46b182b692a0c4fc61ed P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:58.641566 29734 catalog_manager.cc:5484] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: config changed from index -1 to 2, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New cstate: current_term: 1 leader_uuid: "9a4d92bc25374fb3960fde7d10be309d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:13:58.688472 32641 consensus_queue.cc:237] T 28f3f05f0a344cd683cb9e010273f908 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
W20240503 02:13:58.694636 32576 consensus_peers.cc:485] T 28f3f05f0a344cd683cb9e010273f908 P 9a4d92bc25374fb3960fde7d10be309d -> Peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Couldn't send request to peer 16440e365430405fb10366ab70e2b429. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 28f3f05f0a344cd683cb9e010273f908. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:58.694715 30133 raft_consensus.cc:1261] T 28f3f05f0a344cd683cb9e010273f908 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:58.695092 434 raft_consensus.cc:1261] T 28f3f05f0a344cd683cb9e010273f908 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:58.695327 9169 consensus_queue.cc:1035] T 28f3f05f0a344cd683cb9e010273f908 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:58.695700 8726 consensus_queue.cc:1035] T 28f3f05f0a344cd683cb9e010273f908 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:58.706866 8726 raft_consensus.cc:2932] T 28f3f05f0a344cd683cb9e010273f908 P 9a4d92bc25374fb3960fde7d10be309d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:58.707662 30133 raft_consensus.cc:2932] T 28f3f05f0a344cd683cb9e010273f908 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:58.713366 434 raft_consensus.cc:2932] T 28f3f05f0a344cd683cb9e010273f908 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:58.715898 29734 catalog_manager.cc:5484] T 28f3f05f0a344cd683cb9e010273f908 P 9a4d92bc25374fb3960fde7d10be309d reported cstate change: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New cstate: current_term: 1 leader_uuid: "9a4d92bc25374fb3960fde7d10be309d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:13:58.765761 29874 consensus_queue.cc:237] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:58.768167 565 raft_consensus.cc:1261] T 75497fcddb62421395bd0fa7c88a406c P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:58.768260 32641 raft_consensus.cc:1261] T 75497fcddb62421395bd0fa7c88a406c P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:13:58.768538 29809 consensus_peers.cc:485] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 -> Peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Couldn't send request to peer 155735c7f8b14997ac4c770aee2a9401. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 75497fcddb62421395bd0fa7c88a406c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:58.769131 9206 consensus_queue.cc:1035] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:58.769608 8770 consensus_queue.cc:1035] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:58.772244 9206 raft_consensus.cc:2932] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:58.772881 32641 raft_consensus.cc:2932] T 75497fcddb62421395bd0fa7c88a406c P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:58.778694 565 raft_consensus.cc:2932] T 75497fcddb62421395bd0fa7c88a406c P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:58.779487 29725 catalog_manager.cc:5484] T 75497fcddb62421395bd0fa7c88a406c P 9a4d92bc25374fb3960fde7d10be309d reported cstate change: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:13:59.028796 9244 ts_tablet_manager.cc:902] T 5a44b34b585b47528c22fc678289b01a P 16440e365430405fb10366ab70e2b429: Initiating tablet copy from peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:59.035408 9243 ts_tablet_manager.cc:902] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 8a666e798fe8477d8dd1e51b385d5ead: Initiating tablet copy from peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:13:59.045256 9242 ts_tablet_manager.cc:902] T 480f7b57c2c04c1399b945a023fbe41f P 9a4d92bc25374fb3960fde7d10be309d: Initiating tablet copy from peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:13:59.077668 9243 tablet_copy_client.cc:308] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.5:35595
I20240503 02:13:59.078454 32662 tablet_copy_service.cc:140] P 9a4d92bc25374fb3960fde7d10be309d: Received BeginTabletCopySession request for tablet fb8fcfa3f3ab46b182b692a0c4fc61ed from peer 8a666e798fe8477d8dd1e51b385d5ead ({username='slave'} at 127.22.108.6:48381)
I20240503 02:13:59.078629 32662 tablet_copy_service.cc:161] P 9a4d92bc25374fb3960fde7d10be309d: Beginning new tablet copy session on tablet fb8fcfa3f3ab46b182b692a0c4fc61ed from peer 8a666e798fe8477d8dd1e51b385d5ead at {username='slave'} at 127.22.108.6:48381: session id = 8a666e798fe8477d8dd1e51b385d5ead-fb8fcfa3f3ab46b182b692a0c4fc61ed
I20240503 02:13:59.081590 32662 tablet_copy_source_session.cc:205] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 9a4d92bc25374fb3960fde7d10be309d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:59.083138 9243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb8fcfa3f3ab46b182b692a0c4fc61ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:59.087352 9242 tablet_copy_client.cc:308] T 480f7b57c2c04c1399b945a023fbe41f P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.6:39975
I20240503 02:13:59.087994 327 tablet_copy_service.cc:140] P 8a666e798fe8477d8dd1e51b385d5ead: Received BeginTabletCopySession request for tablet 480f7b57c2c04c1399b945a023fbe41f from peer 9a4d92bc25374fb3960fde7d10be309d ({username='slave'} at 127.22.108.5:36377)
I20240503 02:13:59.088106 327 tablet_copy_service.cc:161] P 8a666e798fe8477d8dd1e51b385d5ead: Beginning new tablet copy session on tablet 480f7b57c2c04c1399b945a023fbe41f from peer 9a4d92bc25374fb3960fde7d10be309d at {username='slave'} at 127.22.108.5:36377: session id = 9a4d92bc25374fb3960fde7d10be309d-480f7b57c2c04c1399b945a023fbe41f
I20240503 02:13:59.090415 327 tablet_copy_source_session.cc:205] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:59.092061 9242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 480f7b57c2c04c1399b945a023fbe41f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:59.093255 9244 tablet_copy_client.cc:308] T 5a44b34b585b47528c22fc678289b01a P 16440e365430405fb10366ab70e2b429: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.6:39975
I20240503 02:13:59.093907 327 tablet_copy_service.cc:140] P 8a666e798fe8477d8dd1e51b385d5ead: Received BeginTabletCopySession request for tablet 5a44b34b585b47528c22fc678289b01a from peer 16440e365430405fb10366ab70e2b429 ({username='slave'} at 127.22.108.1:40161)
I20240503 02:13:59.094026 327 tablet_copy_service.cc:161] P 8a666e798fe8477d8dd1e51b385d5ead: Beginning new tablet copy session on tablet 5a44b34b585b47528c22fc678289b01a from peer 16440e365430405fb10366ab70e2b429 at {username='slave'} at 127.22.108.1:40161: session id = 16440e365430405fb10366ab70e2b429-5a44b34b585b47528c22fc678289b01a
I20240503 02:13:59.096648 327 tablet_copy_source_session.cc:205] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:59.098060 9244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a44b34b585b47528c22fc678289b01a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:59.099242 9242 tablet_copy_client.cc:783] T 480f7b57c2c04c1399b945a023fbe41f P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:59.099452 9242 tablet_copy_client.cc:647] T 480f7b57c2c04c1399b945a023fbe41f P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:59.101130 9242 tablet_copy_client.cc:523] T 480f7b57c2c04c1399b945a023fbe41f P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:59.104566 9242 tablet_bootstrap.cc:492] T 480f7b57c2c04c1399b945a023fbe41f P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:13:59.104674 9244 tablet_copy_client.cc:783] T 5a44b34b585b47528c22fc678289b01a P 16440e365430405fb10366ab70e2b429: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:59.104914 9244 tablet_copy_client.cc:647] T 5a44b34b585b47528c22fc678289b01a P 16440e365430405fb10366ab70e2b429: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:59.111310 9243 tablet_copy_client.cc:783] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:59.111557 9243 tablet_copy_client.cc:647] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:59.112409 9244 tablet_copy_client.cc:523] T 5a44b34b585b47528c22fc678289b01a P 16440e365430405fb10366ab70e2b429: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:59.114048 9243 tablet_copy_client.cc:523] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:59.120332 9244 tablet_bootstrap.cc:492] T 5a44b34b585b47528c22fc678289b01a P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:13:59.121600 9242 tablet_bootstrap.cc:492] T 480f7b57c2c04c1399b945a023fbe41f P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:59.124593 9243 tablet_bootstrap.cc:492] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:13:59.129724 9242 tablet_bootstrap.cc:492] T 480f7b57c2c04c1399b945a023fbe41f P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap complete.
I20240503 02:13:59.130024 9242 ts_tablet_manager.cc:1367] T 480f7b57c2c04c1399b945a023fbe41f P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.026s user 0.012s sys 0.003s
I20240503 02:13:59.130872 9242 raft_consensus.cc:348] T 480f7b57c2c04c1399b945a023fbe41f P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:59.131091 9242 raft_consensus.cc:729] T 480f7b57c2c04c1399b945a023fbe41f P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:13:59.131317 9242 consensus_queue.cc:260] T 480f7b57c2c04c1399b945a023fbe41f P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:13:59.136152 9242 ts_tablet_manager.cc:1398] T 480f7b57c2c04c1399b945a023fbe41f P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:13:59.143724 9243 tablet_bootstrap.cc:492] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:59.144240 9243 tablet_bootstrap.cc:492] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap complete.
I20240503 02:13:59.144495 9243 ts_tablet_manager.cc:1367] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.000s
I20240503 02:13:59.145309 9243 raft_consensus.cc:348] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:59.145532 9243 raft_consensus.cc:729] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:13:59.145718 9243 consensus_queue.cc:260] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:13:59.146195 9243 ts_tablet_manager.cc:1398] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:13:59.153683 32662 tablet_copy_service.cc:342] P 9a4d92bc25374fb3960fde7d10be309d: Request end of tablet copy session 8a666e798fe8477d8dd1e51b385d5ead-fb8fcfa3f3ab46b182b692a0c4fc61ed received from {username='slave'} at 127.22.108.6:48381
I20240503 02:13:59.153820 32662 tablet_copy_service.cc:433] P 9a4d92bc25374fb3960fde7d10be309d: ending tablet copy session 8a666e798fe8477d8dd1e51b385d5ead-fb8fcfa3f3ab46b182b692a0c4fc61ed on tablet fb8fcfa3f3ab46b182b692a0c4fc61ed with peer 8a666e798fe8477d8dd1e51b385d5ead
I20240503 02:13:59.158481 327 tablet_copy_service.cc:342] P 8a666e798fe8477d8dd1e51b385d5ead: Request end of tablet copy session 9a4d92bc25374fb3960fde7d10be309d-480f7b57c2c04c1399b945a023fbe41f received from {username='slave'} at 127.22.108.5:36377
I20240503 02:13:59.158598 327 tablet_copy_service.cc:433] P 8a666e798fe8477d8dd1e51b385d5ead: ending tablet copy session 9a4d92bc25374fb3960fde7d10be309d-480f7b57c2c04c1399b945a023fbe41f on tablet 480f7b57c2c04c1399b945a023fbe41f with peer 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:13:59.246803 9270 ts_tablet_manager.cc:902] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429: Initiating tablet copy from peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:13:59.248714 30199 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.3:42679, user_credentials={real_user=slave}} blocked reactor thread for 140952us
I20240503 02:13:59.280184 9275 ts_tablet_manager.cc:902] T 75497fcddb62421395bd0fa7c88a406c P 155735c7f8b14997ac4c770aee2a9401: Initiating tablet copy from peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:13:59.309633 9244 tablet_bootstrap.cc:492] T 5a44b34b585b47528c22fc678289b01a P 16440e365430405fb10366ab70e2b429: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:59.310039 9244 tablet_bootstrap.cc:492] T 5a44b34b585b47528c22fc678289b01a P 16440e365430405fb10366ab70e2b429: Bootstrap complete.
I20240503 02:13:59.310338 9244 ts_tablet_manager.cc:1367] T 5a44b34b585b47528c22fc678289b01a P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.190s user 0.064s sys 0.082s
I20240503 02:13:59.311117 9244 raft_consensus.cc:348] T 5a44b34b585b47528c22fc678289b01a P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:59.311348 9244 raft_consensus.cc:729] T 5a44b34b585b47528c22fc678289b01a P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:13:59.311682 9244 consensus_queue.cc:260] T 5a44b34b585b47528c22fc678289b01a P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:59.313886 9244 ts_tablet_manager.cc:1398] T 5a44b34b585b47528c22fc678289b01a P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:13:59.318754 327 tablet_copy_service.cc:342] P 8a666e798fe8477d8dd1e51b385d5ead: Request end of tablet copy session 16440e365430405fb10366ab70e2b429-5a44b34b585b47528c22fc678289b01a received from {username='slave'} at 127.22.108.1:40161
I20240503 02:13:59.318878 327 tablet_copy_service.cc:433] P 8a666e798fe8477d8dd1e51b385d5ead: ending tablet copy session 16440e365430405fb10366ab70e2b429-5a44b34b585b47528c22fc678289b01a on tablet 5a44b34b585b47528c22fc678289b01a with peer 16440e365430405fb10366ab70e2b429
I20240503 02:13:59.323916 9270 tablet_copy_client.cc:308] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.5:35595
I20240503 02:13:59.324663 32662 tablet_copy_service.cc:140] P 9a4d92bc25374fb3960fde7d10be309d: Received BeginTabletCopySession request for tablet 28f3f05f0a344cd683cb9e010273f908 from peer 16440e365430405fb10366ab70e2b429 ({username='slave'} at 127.22.108.1:43971)
I20240503 02:13:59.324833 32662 tablet_copy_service.cc:161] P 9a4d92bc25374fb3960fde7d10be309d: Beginning new tablet copy session on tablet 28f3f05f0a344cd683cb9e010273f908 from peer 16440e365430405fb10366ab70e2b429 at {username='slave'} at 127.22.108.1:43971: session id = 16440e365430405fb10366ab70e2b429-28f3f05f0a344cd683cb9e010273f908
I20240503 02:13:59.325959 9275 tablet_copy_client.cc:308] T 75497fcddb62421395bd0fa7c88a406c P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.1:36719
I20240503 02:13:59.329102 32662 tablet_copy_source_session.cc:205] T 28f3f05f0a344cd683cb9e010273f908 P 9a4d92bc25374fb3960fde7d10be309d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:59.331264 9270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28f3f05f0a344cd683cb9e010273f908. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:59.339020 9270 tablet_copy_client.cc:783] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:59.339362 9270 tablet_copy_client.cc:647] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:59.345147 29895 tablet_copy_service.cc:140] P 16440e365430405fb10366ab70e2b429: Received BeginTabletCopySession request for tablet 75497fcddb62421395bd0fa7c88a406c from peer 155735c7f8b14997ac4c770aee2a9401 ({username='slave'} at 127.22.108.2:57635)
I20240503 02:13:59.345326 29895 tablet_copy_service.cc:161] P 16440e365430405fb10366ab70e2b429: Beginning new tablet copy session on tablet 75497fcddb62421395bd0fa7c88a406c from peer 155735c7f8b14997ac4c770aee2a9401 at {username='slave'} at 127.22.108.2:57635: session id = 155735c7f8b14997ac4c770aee2a9401-75497fcddb62421395bd0fa7c88a406c
I20240503 02:13:59.347996 9270 tablet_copy_client.cc:523] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:59.348636 29895 tablet_copy_source_session.cc:205] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:59.354254 9270 tablet_bootstrap.cc:492] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:13:59.361900 9275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75497fcddb62421395bd0fa7c88a406c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:59.394047 9270 tablet_bootstrap.cc:492] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:59.394546 9270 tablet_bootstrap.cc:492] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429: Bootstrap complete.
I20240503 02:13:59.399242 9270 ts_tablet_manager.cc:1367] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.005s
I20240503 02:13:59.407404 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:13:59.407732 9270 raft_consensus.cc:348] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:59.408047 9270 raft_consensus.cc:729] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:13:59.408332 9270 consensus_queue.cc:260] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:13:59.410455 9270 ts_tablet_manager.cc:1398] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20240503 02:13:59.411460 9275 tablet_copy_client.cc:783] T 75497fcddb62421395bd0fa7c88a406c P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:59.411726 9275 tablet_copy_client.cc:647] T 75497fcddb62421395bd0fa7c88a406c P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:59.412232 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:59.418505 32662 tablet_copy_service.cc:342] P 9a4d92bc25374fb3960fde7d10be309d: Request end of tablet copy session 16440e365430405fb10366ab70e2b429-28f3f05f0a344cd683cb9e010273f908 received from {username='slave'} at 127.22.108.1:43971
I20240503 02:13:59.418644 32662 tablet_copy_service.cc:433] P 9a4d92bc25374fb3960fde7d10be309d: ending tablet copy session 16440e365430405fb10366ab70e2b429-28f3f05f0a344cd683cb9e010273f908 on tablet 28f3f05f0a344cd683cb9e010273f908 with peer 16440e365430405fb10366ab70e2b429
I20240503 02:13:59.453100 9275 tablet_copy_client.cc:523] T 75497fcddb62421395bd0fa7c88a406c P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:59.458310 307 raft_consensus.cc:1203] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:59.477186 9275 tablet_bootstrap.cc:492] T 75497fcddb62421395bd0fa7c88a406c P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:13:59.478329 30244 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:59.495111 32640 raft_consensus.cc:1203] T 480f7b57c2c04c1399b945a023fbe41f P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:59.520587 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:59.557098 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:59.565469 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:59.586481 29874 raft_consensus.cc:1203] T 5a44b34b585b47528c22fc678289b01a P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:59.662459 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:13:59.672921 9275 tablet_bootstrap.cc:492] T 75497fcddb62421395bd0fa7c88a406c P 155735c7f8b14997ac4c770aee2a9401: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:59.673359 9275 tablet_bootstrap.cc:492] T 75497fcddb62421395bd0fa7c88a406c P 155735c7f8b14997ac4c770aee2a9401: Bootstrap complete.
I20240503 02:13:59.673633 9275 ts_tablet_manager.cc:1367] T 75497fcddb62421395bd0fa7c88a406c P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.197s user 0.022s sys 0.024s
I20240503 02:13:59.674373 9275 raft_consensus.cc:348] T 75497fcddb62421395bd0fa7c88a406c P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:59.674589 9275 raft_consensus.cc:729] T 75497fcddb62421395bd0fa7c88a406c P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:13:59.674801 9275 consensus_queue.cc:260] T 75497fcddb62421395bd0fa7c88a406c P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:13:59.695427 9275 ts_tablet_manager.cc:1398] T 75497fcddb62421395bd0fa7c88a406c P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240503 02:13:59.705592 29895 tablet_copy_service.cc:342] P 16440e365430405fb10366ab70e2b429: Request end of tablet copy session 155735c7f8b14997ac4c770aee2a9401-75497fcddb62421395bd0fa7c88a406c received from {username='slave'} at 127.22.108.2:57635
I20240503 02:13:59.705746 29895 tablet_copy_service.cc:433] P 16440e365430405fb10366ab70e2b429: ending tablet copy session 155735c7f8b14997ac4c770aee2a9401-75497fcddb62421395bd0fa7c88a406c on tablet 75497fcddb62421395bd0fa7c88a406c with peer 155735c7f8b14997ac4c770aee2a9401
I20240503 02:13:59.730518 29874 raft_consensus.cc:1203] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:59.754206 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:13:59.854588 30004 raft_consensus.cc:1203] T 75497fcddb62421395bd0fa7c88a406c P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:14:00.287211 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:00.339385 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:00.347452 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:00.347847 30244 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:14:00.348234 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:00.384922 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:00.399359 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:00.500703 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:00.610270 32640 consensus_queue.cc:237] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:00.612646 307 raft_consensus.cc:1261] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:00.612790 30265 raft_consensus.cc:1261] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:00.613284 8726 consensus_queue.cc:1035] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:00.613520 9169 consensus_queue.cc:1035] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:00.616555 8726 raft_consensus.cc:2932] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 9a4d92bc25374fb3960fde7d10be309d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:00.617971 30265 raft_consensus.cc:2932] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:00.618274 307 raft_consensus.cc:2932] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:00.624723 29726 catalog_manager.cc:5484] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New cstate: current_term: 1 leader_uuid: "9a4d92bc25374fb3960fde7d10be309d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:00.629928 547 tablet_service.cc:1514] Processing DeleteTablet for tablet fb8fcfa3f3ab46b182b692a0c4fc61ed with delete_type TABLET_DATA_TOMBSTONED (TS f575ce92c0224eeb80b9fae5f6561fbe not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:39028
I20240503 02:14:00.638720 9393 tablet_replica.cc:330] T fb8fcfa3f3ab46b182b692a0c4fc61ed P f575ce92c0224eeb80b9fae5f6561fbe: stopping tablet replica
I20240503 02:14:00.638923 9393 raft_consensus.cc:2229] T fb8fcfa3f3ab46b182b692a0c4fc61ed P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:00.639875 9393 raft_consensus.cc:2258] T fb8fcfa3f3ab46b182b692a0c4fc61ed P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:00.640614 9393 ts_tablet_manager.cc:1875] T fb8fcfa3f3ab46b182b692a0c4fc61ed P f575ce92c0224eeb80b9fae5f6561fbe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:00.647920 9393 ts_tablet_manager.cc:1888] T fb8fcfa3f3ab46b182b692a0c4fc61ed P f575ce92c0224eeb80b9fae5f6561fbe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:14:00.657167 9393 log.cc:1196] T fb8fcfa3f3ab46b182b692a0c4fc61ed P f575ce92c0224eeb80b9fae5f6561fbe: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal/wals/fb8fcfa3f3ab46b182b692a0c4fc61ed
I20240503 02:14:00.658951 29718 catalog_manager.cc:4830] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): tablet fb8fcfa3f3ab46b182b692a0c4fc61ed (table rebalancer_extra_table_30 [id=854d24d23fe34d699e6656ef51d15693]) successfully deleted
I20240503 02:14:00.912724 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:00.971105 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:00.990325 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:01.024618 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:01.048872 30244 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:14:01.052045 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240503 02:14:01.097988 371 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.2:37873, user_credentials={real_user=slave}} blocked reactor thread for 65995.1us
I20240503 02:14:01.103684 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:01.143822 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:01.279500 306 consensus_queue.cc:237] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:14:01.281702 32640 raft_consensus.cc:1261] T 480f7b57c2c04c1399b945a023fbe41f P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:01.281754 30005 raft_consensus.cc:1261] T 480f7b57c2c04c1399b945a023fbe41f P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:01.284001 9456 consensus_queue.cc:1035] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:01.284610 9127 consensus_queue.cc:1035] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:01.286806 9456 raft_consensus.cc:2932] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:01.287415 32641 raft_consensus.cc:2932] T 480f7b57c2c04c1399b945a023fbe41f P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:01.287644 30005 raft_consensus.cc:2932] T 480f7b57c2c04c1399b945a023fbe41f P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:01.294458 29733 catalog_manager.cc:5484] T 480f7b57c2c04c1399b945a023fbe41f P 9a4d92bc25374fb3960fde7d10be309d reported cstate change: config changed from index 2 to 3, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) evicted. New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:01.302999 30244 tablet_service.cc:1514] Processing DeleteTablet for tablet 480f7b57c2c04c1399b945a023fbe41f with delete_type TABLET_DATA_TOMBSTONED (TS 1f10f80f13c84e80b03eda7085a3b012 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:49856
I20240503 02:14:01.303642 9480 tablet_replica.cc:330] T 480f7b57c2c04c1399b945a023fbe41f P 1f10f80f13c84e80b03eda7085a3b012: stopping tablet replica
I20240503 02:14:01.303967 9480 raft_consensus.cc:2229] T 480f7b57c2c04c1399b945a023fbe41f P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:01.304345 9480 raft_consensus.cc:2258] T 480f7b57c2c04c1399b945a023fbe41f P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:01.305086 9480 ts_tablet_manager.cc:1875] T 480f7b57c2c04c1399b945a023fbe41f P 1f10f80f13c84e80b03eda7085a3b012: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:01.311730 9480 ts_tablet_manager.cc:1888] T 480f7b57c2c04c1399b945a023fbe41f P 1f10f80f13c84e80b03eda7085a3b012: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:14:01.311846 9480 log.cc:1196] T 480f7b57c2c04c1399b945a023fbe41f P 1f10f80f13c84e80b03eda7085a3b012: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/wals/480f7b57c2c04c1399b945a023fbe41f
I20240503 02:14:01.312600 29719 catalog_manager.cc:4830] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): tablet 480f7b57c2c04c1399b945a023fbe41f (table rebalancer_extra_table_46 [id=95e60300d67c446987cfb1f3dbeecf3a]) successfully deleted
I20240503 02:14:01.599550 30244 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:14:01.621260 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:01.641341 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:01.643940 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:01.667407 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:01.686535 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:01.707815 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:01.743423 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:01.884009 306 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5a44b34b585b47528c22fc678289b01a"
dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38966
I20240503 02:14:01.884213 306 raft_consensus.cc:595] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Received request to transfer leadership
I20240503 02:14:01.995632 9456 raft_consensus.cc:979] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead: : Instructing follower 1f10f80f13c84e80b03eda7085a3b012 to start an election
I20240503 02:14:01.995836 9456 raft_consensus.cc:1067] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Signalling peer 1f10f80f13c84e80b03eda7085a3b012 to start an election
I20240503 02:14:01.996451 30265 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5a44b34b585b47528c22fc678289b01a"
dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
from {username='slave'} at 127.22.108.6:54483
I20240503 02:14:01.996635 30265 raft_consensus.cc:482] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:14:01.996724 30265 raft_consensus.cc:3037] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:01.999504 30265 raft_consensus.cc:504] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:14:02.000921 565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a44b34b585b47528c22fc678289b01a" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:14:02.000932 306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a44b34b585b47528c22fc678289b01a" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:14:02.001139 565 raft_consensus.cc:3037] T 5a44b34b585b47528c22fc678289b01a P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:02.001158 306 raft_consensus.cc:3032] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:14:02.001220 306 raft_consensus.cc:729] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Running, Role: LEADER
I20240503 02:14:02.001506 306 consensus_queue.cc:260] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:14:02.002110 306 raft_consensus.cc:3037] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:02.003120 30265 leader_election.cc:288] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 2 election: Requested vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:14:02.003818 565 raft_consensus.cc:2454] T 5a44b34b585b47528c22fc678289b01a P f575ce92c0224eeb80b9fae5f6561fbe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 2.
I20240503 02:14:02.008121 29874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a44b34b585b47528c22fc678289b01a" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:14:02.008325 29874 raft_consensus.cc:3037] T 5a44b34b585b47528c22fc678289b01a P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:02.009846 306 raft_consensus.cc:2454] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 2.
I20240503 02:14:02.010530 30200 leader_election.cc:302] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, 8a666e798fe8477d8dd1e51b385d5ead, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:14:02.010813 29874 raft_consensus.cc:2454] T 5a44b34b585b47528c22fc678289b01a P 16440e365430405fb10366ab70e2b429 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 2.
I20240503 02:14:02.011178 9264 raft_consensus.cc:2781] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:14:02.011348 9264 raft_consensus.cc:686] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 [term 2 LEADER]: Becoming Leader. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Running, Role: LEADER
I20240503 02:14:02.011677 9264 consensus_queue.cc:237] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:14:02.018085 29726 catalog_manager.cc:5484] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: term changed from 1 to 2, leader changed from 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) to 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4). New cstate: current_term: 2 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:02.198314 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:02.214923 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:02.229568 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:02.246143 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:02.295819 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:02.315166 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:02.354379 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:02.370126 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:02.445683 29873 raft_consensus.cc:1261] T 5a44b34b585b47528c22fc678289b01a P 16440e365430405fb10366ab70e2b429 [term 2 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:14:02.447227 9051 consensus_queue.cc:1035] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:02.471535 565 raft_consensus.cc:1261] T 5a44b34b585b47528c22fc678289b01a P f575ce92c0224eeb80b9fae5f6561fbe [term 2 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:14:02.472095 9051 consensus_queue.cc:1035] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:02.487892 306 raft_consensus.cc:1261] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:14:02.488510 9051 consensus_queue.cc:1035] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:02.858515 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:02.885161 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:02.904258 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:02.912922 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:02.916945 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:02.925266 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:02.970957 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:03.024196 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:03.201715 434 consensus_queue.cc:237] T 5ccde7688c4e49b5b9d89d6e048c308e P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:14:03.203542 29874 raft_consensus.cc:1261] T 5ccde7688c4e49b5b9d89d6e048c308e P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:03.203593 307 raft_consensus.cc:1261] T 5ccde7688c4e49b5b9d89d6e048c308e P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:03.204104 8578 consensus_queue.cc:1035] T 5ccde7688c4e49b5b9d89d6e048c308e P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:03.204425 9451 consensus_queue.cc:1035] T 5ccde7688c4e49b5b9d89d6e048c308e P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:03.208544 9451 raft_consensus.cc:2932] T 5ccde7688c4e49b5b9d89d6e048c308e P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:03.208880 29874 raft_consensus.cc:2932] T 5ccde7688c4e49b5b9d89d6e048c308e P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:03.213996 29731 catalog_manager.cc:5484] T 5ccde7688c4e49b5b9d89d6e048c308e P 16440e365430405fb10366ab70e2b429 reported cstate change: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:03.220314 307 raft_consensus.cc:2932] T 5ccde7688c4e49b5b9d89d6e048c308e P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:03.232419 32619 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ccde7688c4e49b5b9d89d6e048c308e with delete_type TABLET_DATA_TOMBSTONED (TS 9a4d92bc25374fb3960fde7d10be309d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:38200
I20240503 02:14:03.233000 9712 tablet_replica.cc:330] T 5ccde7688c4e49b5b9d89d6e048c308e P 9a4d92bc25374fb3960fde7d10be309d: stopping tablet replica
I20240503 02:14:03.233299 9712 raft_consensus.cc:2229] T 5ccde7688c4e49b5b9d89d6e048c308e P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:03.233539 9712 raft_consensus.cc:2258] T 5ccde7688c4e49b5b9d89d6e048c308e P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:03.234277 9712 ts_tablet_manager.cc:1875] T 5ccde7688c4e49b5b9d89d6e048c308e P 9a4d92bc25374fb3960fde7d10be309d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:03.241086 9712 ts_tablet_manager.cc:1888] T 5ccde7688c4e49b5b9d89d6e048c308e P 9a4d92bc25374fb3960fde7d10be309d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:14:03.241308 9712 log.cc:1196] T 5ccde7688c4e49b5b9d89d6e048c308e P 9a4d92bc25374fb3960fde7d10be309d: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal/wals/5ccde7688c4e49b5b9d89d6e048c308e
I20240503 02:14:03.242103 29719 catalog_manager.cc:4830] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): tablet 5ccde7688c4e49b5b9d89d6e048c308e (table rebalancer_extra_table_2 [id=27940b10e50e4a418d9e7155f5a25394]) successfully deleted
I20240503 02:14:03.550019 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:03.559142 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:03.567572 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:03.595220 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:03.613678 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:03.636230 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:03.663059 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:03.694901 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:03.820129 29873 consensus_queue.cc:237] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:14:03.822775 30005 raft_consensus.cc:1261] T 75497fcddb62421395bd0fa7c88a406c P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:03.823256 32641 raft_consensus.cc:1261] T 75497fcddb62421395bd0fa7c88a406c P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:03.829362 9206 consensus_queue.cc:1035] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:03.830010 9675 consensus_queue.cc:1035] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:03.845577 9808 raft_consensus.cc:2932] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:14:03.846156 30004 raft_consensus.cc:2932] T 75497fcddb62421395bd0fa7c88a406c P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:14:03.846627 32641 raft_consensus.cc:2932] T 75497fcddb62421395bd0fa7c88a406c P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:14:03.853253 29731 catalog_manager.cc:5484] T 75497fcddb62421395bd0fa7c88a406c P 9a4d92bc25374fb3960fde7d10be309d reported cstate change: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:14:03.887606 547 tablet_service.cc:1514] Processing DeleteTablet for tablet 75497fcddb62421395bd0fa7c88a406c with delete_type TABLET_DATA_TOMBSTONED (TS f575ce92c0224eeb80b9fae5f6561fbe not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:39028
I20240503 02:14:03.894824 9819 tablet_replica.cc:330] T 75497fcddb62421395bd0fa7c88a406c P f575ce92c0224eeb80b9fae5f6561fbe: stopping tablet replica
I20240503 02:14:03.895059 9819 raft_consensus.cc:2229] T 75497fcddb62421395bd0fa7c88a406c P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:03.895346 9819 raft_consensus.cc:2258] T 75497fcddb62421395bd0fa7c88a406c P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:03.896127 9819 ts_tablet_manager.cc:1875] T 75497fcddb62421395bd0fa7c88a406c P f575ce92c0224eeb80b9fae5f6561fbe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:03.903043 9819 ts_tablet_manager.cc:1888] T 75497fcddb62421395bd0fa7c88a406c P f575ce92c0224eeb80b9fae5f6561fbe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:14:03.903170 9819 log.cc:1196] T 75497fcddb62421395bd0fa7c88a406c P f575ce92c0224eeb80b9fae5f6561fbe: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal/wals/75497fcddb62421395bd0fa7c88a406c
I20240503 02:14:03.903995 29718 catalog_manager.cc:4830] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): tablet 75497fcddb62421395bd0fa7c88a406c (table rebalancer_extra_table_20 [id=9fae481dc13a491db2520bd2d31be634]) successfully deleted
I20240503 02:14:04.281374 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:04.309834 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:04.335888 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:04.336301 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:04.360146 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:04.360581 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:04.374639 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:04.423427 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:04.563710 30133 consensus_queue.cc:237] T db0756f0f331402aaef0583e1f9e5318 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:04.565672 306 raft_consensus.cc:1261] T db0756f0f331402aaef0583e1f9e5318 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:14:04.565851 30264 raft_consensus.cc:1261] T db0756f0f331402aaef0583e1f9e5318 P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:14:04.566383 9682 consensus_queue.cc:1035] T db0756f0f331402aaef0583e1f9e5318 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:14:04.566687 9889 consensus_queue.cc:1035] T db0756f0f331402aaef0583e1f9e5318 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:14:04.569494 9889 raft_consensus.cc:2932] T db0756f0f331402aaef0583e1f9e5318 P a9094240860e43bfb129193f3bfd5be6 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:04.569864 306 raft_consensus.cc:2932] T db0756f0f331402aaef0583e1f9e5318 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:04.575908 29731 catalog_manager.cc:5484] T db0756f0f331402aaef0583e1f9e5318 P a9094240860e43bfb129193f3bfd5be6 reported cstate change: config changed from index 4 to 6, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) evicted. New cstate: current_term: 2 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:04.580407 29985 tablet_service.cc:1514] Processing DeleteTablet for tablet db0756f0f331402aaef0583e1f9e5318 with delete_type TABLET_DATA_TOMBSTONED (TS 155735c7f8b14997ac4c770aee2a9401 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52158
I20240503 02:14:04.581460 9912 tablet_replica.cc:330] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:14:04.581813 9912 raft_consensus.cc:2229] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:04.582064 9912 raft_consensus.cc:2258] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:04.583050 9912 ts_tablet_manager.cc:1875] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:04.590641 9912 ts_tablet_manager.cc:1888] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 02:14:04.590946 9912 log.cc:1196] T db0756f0f331402aaef0583e1f9e5318 P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/db0756f0f331402aaef0583e1f9e5318
I20240503 02:14:04.592145 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet db0756f0f331402aaef0583e1f9e5318 (table rebalance_test_table_13 [id=df07c1fd4e654fea9a59ea9a0ea79595]) successfully deleted
I20240503 02:14:04.597748 30265 raft_consensus.cc:2932] T db0756f0f331402aaef0583e1f9e5318 P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:04.666190 306 consensus_queue.cc:237] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:14:04.669099 565 raft_consensus.cc:1261] T 3c437d9d466348f2863c407666c348aa P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:04.669283 436 raft_consensus.cc:1261] T 3c437d9d466348f2863c407666c348aa P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:04.669739 9825 consensus_queue.cc:1035] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:14:04.670008 32709 consensus_peers.cc:485] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): Couldn't send request to peer 1f10f80f13c84e80b03eda7085a3b012. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3c437d9d466348f2863c407666c348aa. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:04.670270 9825 consensus_queue.cc:1035] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:04.674146 9944 raft_consensus.cc:2932] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:14:04.674832 565 raft_consensus.cc:2932] T 3c437d9d466348f2863c407666c348aa P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:14:04.681149 436 raft_consensus.cc:2932] T 3c437d9d466348f2863c407666c348aa P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:14:04.681958 29731 catalog_manager.cc:5484] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:14:04.723842 29874 consensus_queue.cc:237] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
W20240503 02:14:04.735030 29809 consensus_peers.cc:485] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f00949b4ca84a49b0f6e62bd7373586. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:04.735220 30004 raft_consensus.cc:1261] T 3f00949b4ca84a49b0f6e62bd7373586 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:14:04.735369 30133 raft_consensus.cc:1261] T 3f00949b4ca84a49b0f6e62bd7373586 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:14:04.735767 9963 consensus_queue.cc:1035] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:14:04.736107 9206 consensus_queue.cc:1035] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:14:04.739635 9963 raft_consensus.cc:2932] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:04.740451 30004 raft_consensus.cc:2932] T 3f00949b4ca84a49b0f6e62bd7373586 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:04.749764 29731 catalog_manager.cc:5484] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 reported cstate change: config changed from index -1 to 4, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:04.751987 30132 raft_consensus.cc:2932] T 3f00949b4ca84a49b0f6e62bd7373586 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:04.793234 30132 consensus_queue.cc:237] T 5a5d5658fe5e4104a15b404cb2022b7c P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:14:04.795984 30004 raft_consensus.cc:1261] T 5a5d5658fe5e4104a15b404cb2022b7c P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:04.796173 565 raft_consensus.cc:1261] T 5a5d5658fe5e4104a15b404cb2022b7c P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:14:04.796196 30070 consensus_peers.cc:485] T 5a5d5658fe5e4104a15b404cb2022b7c P a9094240860e43bfb129193f3bfd5be6 -> Peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): Couldn't send request to peer fbbc2be6382e47908ecb7c68c367bd26. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5a5d5658fe5e4104a15b404cb2022b7c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:04.796512 9682 consensus_queue.cc:1035] T 5a5d5658fe5e4104a15b404cb2022b7c P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:04.796769 8477 consensus_queue.cc:1035] T 5a5d5658fe5e4104a15b404cb2022b7c P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:04.799641 9682 raft_consensus.cc:2932] T 5a5d5658fe5e4104a15b404cb2022b7c P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:14:04.800261 565 raft_consensus.cc:2932] T 5a5d5658fe5e4104a15b404cb2022b7c P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:14:04.803522 30004 raft_consensus.cc:2932] T 5a5d5658fe5e4104a15b404cb2022b7c P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:14:04.805670 29731 catalog_manager.cc:5484] T 5a5d5658fe5e4104a15b404cb2022b7c P a9094240860e43bfb129193f3bfd5be6 reported cstate change: config changed from index -1 to 2, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:14:04.848605 30132 consensus_queue.cc:237] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:14:04.851218 29874 raft_consensus.cc:1261] T e80b43971dc246d9bfd5e1c603f8415c P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:14:04.852068 30070 consensus_peers.cc:485] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e80b43971dc246d9bfd5e1c603f8415c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:04.852329 30265 raft_consensus.cc:1261] T e80b43971dc246d9bfd5e1c603f8415c P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:14:04.852795 8477 consensus_queue.cc:1035] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:14:04.854020 8477 consensus_queue.cc:1035] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:14:04.866699 8477 raft_consensus.cc:2932] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:04.870378 29874 raft_consensus.cc:2932] T e80b43971dc246d9bfd5e1c603f8415c P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:04.872253 30264 raft_consensus.cc:2932] T e80b43971dc246d9bfd5e1c603f8415c P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:04.876307 29731 catalog_manager.cc:5484] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 reported cstate change: config changed from index -1 to 4, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:04.954144 30004 consensus_queue.cc:237] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:04.959198 30264 raft_consensus.cc:1261] T 6635aacafed54994965ad48b2df4f163 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:14:04.959717 29940 consensus_peers.cc:485] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): Couldn't send request to peer 8a666e798fe8477d8dd1e51b385d5ead. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6635aacafed54994965ad48b2df4f163. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:04.959995 8698 consensus_queue.cc:1035] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:14:04.960494 30133 raft_consensus.cc:1261] T 6635aacafed54994965ad48b2df4f163 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:14:04.962280 8698 consensus_queue.cc:1035] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:14:04.978130 9887 raft_consensus.cc:2932] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:04.979688 30133 raft_consensus.cc:2932] T 6635aacafed54994965ad48b2df4f163 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:04.986971 30265 raft_consensus.cc:2932] T 6635aacafed54994965ad48b2df4f163 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:04.996980 29726 catalog_manager.cc:5484] T 6635aacafed54994965ad48b2df4f163 P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: config changed from index -1 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:05.077168 32640 consensus_queue.cc:237] T eec091fa0ba947d68577d562e538ab56 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:05.079521 436 raft_consensus.cc:1261] T eec091fa0ba947d68577d562e538ab56 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:05.080086 10040 consensus_queue.cc:1035] T eec091fa0ba947d68577d562e538ab56 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:14:05.080363 32574 consensus_peers.cc:485] T eec091fa0ba947d68577d562e538ab56 P 9a4d92bc25374fb3960fde7d10be309d -> Peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): Couldn't send request to peer 8a666e798fe8477d8dd1e51b385d5ead. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eec091fa0ba947d68577d562e538ab56. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:05.080750 565 raft_consensus.cc:1261] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:05.081277 9862 consensus_queue.cc:1035] T eec091fa0ba947d68577d562e538ab56 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:05.083756 10040 raft_consensus.cc:2932] T eec091fa0ba947d68577d562e538ab56 P 9a4d92bc25374fb3960fde7d10be309d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:05.084352 565 raft_consensus.cc:2932] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:05.090716 29731 catalog_manager.cc:5484] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: config changed from index -1 to 2, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New cstate: current_term: 1 leader_uuid: "9a4d92bc25374fb3960fde7d10be309d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:05.096693 436 raft_consensus.cc:2932] T eec091fa0ba947d68577d562e538ab56 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:05.117082 10054 ts_tablet_manager.cc:902] T 3c437d9d466348f2863c407666c348aa P 1f10f80f13c84e80b03eda7085a3b012: Initiating tablet copy from peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:14:05.127369 30264 consensus_queue.cc:237] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:14:05.130626 30133 raft_consensus.cc:1261] T c19017f829a74cc2b6dc41a2a25a3831 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:14:05.130667 30199 consensus_peers.cc:485] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c19017f829a74cc2b6dc41a2a25a3831. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:05.130793 29874 raft_consensus.cc:1261] T c19017f829a74cc2b6dc41a2a25a3831 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:14:05.131474 10002 consensus_queue.cc:1035] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:14:05.131815 9264 consensus_queue.cc:1035] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:14:05.142510 10002 raft_consensus.cc:2932] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:05.142910 29873 raft_consensus.cc:2932] T c19017f829a74cc2b6dc41a2a25a3831 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:05.151538 29725 catalog_manager.cc:5484] T c19017f829a74cc2b6dc41a2a25a3831 P 16440e365430405fb10366ab70e2b429 reported cstate change: config changed from index -1 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:05.181739 10054 tablet_copy_client.cc:308] T 3c437d9d466348f2863c407666c348aa P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.6:39975
I20240503 02:14:05.181354 30133 raft_consensus.cc:2932] T c19017f829a74cc2b6dc41a2a25a3831 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:05.182505 327 tablet_copy_service.cc:140] P 8a666e798fe8477d8dd1e51b385d5ead: Received BeginTabletCopySession request for tablet 3c437d9d466348f2863c407666c348aa from peer 1f10f80f13c84e80b03eda7085a3b012 ({username='slave'} at 127.22.108.4:53319)
I20240503 02:14:05.182636 327 tablet_copy_service.cc:161] P 8a666e798fe8477d8dd1e51b385d5ead: Beginning new tablet copy session on tablet 3c437d9d466348f2863c407666c348aa from peer 1f10f80f13c84e80b03eda7085a3b012 at {username='slave'} at 127.22.108.4:53319: session id = 1f10f80f13c84e80b03eda7085a3b012-3c437d9d466348f2863c407666c348aa
I20240503 02:14:05.185439 327 tablet_copy_source_session.cc:205] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:05.187096 10054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c437d9d466348f2863c407666c348aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:05.195376 10054 tablet_copy_client.cc:783] T 3c437d9d466348f2863c407666c348aa P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:05.196060 10054 tablet_copy_client.cc:647] T 3c437d9d466348f2863c407666c348aa P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:05.198837 10054 tablet_copy_client.cc:523] T 3c437d9d466348f2863c407666c348aa P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:05.202653 10054 tablet_bootstrap.cc:492] T 3c437d9d466348f2863c407666c348aa P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:14:05.209990 10054 tablet_bootstrap.cc:492] T 3c437d9d466348f2863c407666c348aa P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:05.210495 10054 tablet_bootstrap.cc:492] T 3c437d9d466348f2863c407666c348aa P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap complete.
I20240503 02:14:05.210791 10054 ts_tablet_manager.cc:1367] T 3c437d9d466348f2863c407666c348aa P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20240503 02:14:05.211493 10054 raft_consensus.cc:348] T 3c437d9d466348f2863c407666c348aa P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:14:05.214772 10054 raft_consensus.cc:729] T 3c437d9d466348f2863c407666c348aa P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:14:05.215055 10054 consensus_queue.cc:260] T 3c437d9d466348f2863c407666c348aa P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:14:05.215636 10054 ts_tablet_manager.cc:1398] T 3c437d9d466348f2863c407666c348aa P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 02:14:05.219672 327 tablet_copy_service.cc:342] P 8a666e798fe8477d8dd1e51b385d5ead: Request end of tablet copy session 1f10f80f13c84e80b03eda7085a3b012-3c437d9d466348f2863c407666c348aa received from {username='slave'} at 127.22.108.4:53319
I20240503 02:14:05.219785 327 tablet_copy_service.cc:433] P 8a666e798fe8477d8dd1e51b385d5ead: ending tablet copy session 1f10f80f13c84e80b03eda7085a3b012-3c437d9d466348f2863c407666c348aa on tablet 3c437d9d466348f2863c407666c348aa with peer 1f10f80f13c84e80b03eda7085a3b012
I20240503 02:14:05.237849 10080 ts_tablet_manager.cc:902] T 5a5d5658fe5e4104a15b404cb2022b7c P fbbc2be6382e47908ecb7c68c367bd26: Initiating tablet copy from peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:14:05.238981 10084 ts_tablet_manager.cc:902] T 3f00949b4ca84a49b0f6e62bd7373586 P 9a4d92bc25374fb3960fde7d10be309d: Initiating tablet copy from peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:14:05.259195 10080 tablet_copy_client.cc:308] T 5a5d5658fe5e4104a15b404cb2022b7c P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.3:42679
I20240503 02:14:05.260344 30156 tablet_copy_service.cc:140] P a9094240860e43bfb129193f3bfd5be6: Received BeginTabletCopySession request for tablet 5a5d5658fe5e4104a15b404cb2022b7c from peer fbbc2be6382e47908ecb7c68c367bd26 ({username='slave'} at 127.22.108.7:37671)
I20240503 02:14:05.260541 30156 tablet_copy_service.cc:161] P a9094240860e43bfb129193f3bfd5be6: Beginning new tablet copy session on tablet 5a5d5658fe5e4104a15b404cb2022b7c from peer fbbc2be6382e47908ecb7c68c367bd26 at {username='slave'} at 127.22.108.7:37671: session id = fbbc2be6382e47908ecb7c68c367bd26-5a5d5658fe5e4104a15b404cb2022b7c
I20240503 02:14:05.263533 10084 tablet_copy_client.cc:308] T 3f00949b4ca84a49b0f6e62bd7373586 P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.1:36719
I20240503 02:14:05.264691 29895 tablet_copy_service.cc:140] P 16440e365430405fb10366ab70e2b429: Received BeginTabletCopySession request for tablet 3f00949b4ca84a49b0f6e62bd7373586 from peer 9a4d92bc25374fb3960fde7d10be309d ({username='slave'} at 127.22.108.5:49385)
I20240503 02:14:05.264853 29895 tablet_copy_service.cc:161] P 16440e365430405fb10366ab70e2b429: Beginning new tablet copy session on tablet 3f00949b4ca84a49b0f6e62bd7373586 from peer 9a4d92bc25374fb3960fde7d10be309d at {username='slave'} at 127.22.108.5:49385: session id = 9a4d92bc25374fb3960fde7d10be309d-3f00949b4ca84a49b0f6e62bd7373586
I20240503 02:14:05.266610 30156 tablet_copy_source_session.cc:205] T 5a5d5658fe5e4104a15b404cb2022b7c P a9094240860e43bfb129193f3bfd5be6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:05.267877 29895 tablet_copy_source_session.cc:205] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:05.269580 10080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a5d5658fe5e4104a15b404cb2022b7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:05.270490 10084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f00949b4ca84a49b0f6e62bd7373586. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:05.275991 10080 tablet_copy_client.cc:783] T 5a5d5658fe5e4104a15b404cb2022b7c P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:05.276266 10080 tablet_copy_client.cc:647] T 5a5d5658fe5e4104a15b404cb2022b7c P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:05.277822 10080 tablet_copy_client.cc:523] T 5a5d5658fe5e4104a15b404cb2022b7c P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:05.280059 10084 tablet_copy_client.cc:783] T 3f00949b4ca84a49b0f6e62bd7373586 P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:05.281092 10080 tablet_bootstrap.cc:492] T 5a5d5658fe5e4104a15b404cb2022b7c P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:14:05.282284 10084 tablet_copy_client.cc:647] T 3f00949b4ca84a49b0f6e62bd7373586 P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:05.285003 10084 tablet_copy_client.cc:523] T 3f00949b4ca84a49b0f6e62bd7373586 P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:05.287635 10080 tablet_bootstrap.cc:492] T 5a5d5658fe5e4104a15b404cb2022b7c P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:05.287993 10080 tablet_bootstrap.cc:492] T 5a5d5658fe5e4104a15b404cb2022b7c P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap complete.
I20240503 02:14:05.288201 10080 ts_tablet_manager.cc:1367] T 5a5d5658fe5e4104a15b404cb2022b7c P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20240503 02:14:05.288424 10084 tablet_bootstrap.cc:492] T 3f00949b4ca84a49b0f6e62bd7373586 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:14:05.288877 10080 raft_consensus.cc:348] T 5a5d5658fe5e4104a15b404cb2022b7c P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:14:05.289125 10080 raft_consensus.cc:729] T 5a5d5658fe5e4104a15b404cb2022b7c P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:14:05.289353 10080 consensus_queue.cc:260] T 5a5d5658fe5e4104a15b404cb2022b7c P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:14:05.289901 10080 ts_tablet_manager.cc:1398] T 5a5d5658fe5e4104a15b404cb2022b7c P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:14:05.290619 30156 tablet_copy_service.cc:342] P a9094240860e43bfb129193f3bfd5be6: Request end of tablet copy session fbbc2be6382e47908ecb7c68c367bd26-5a5d5658fe5e4104a15b404cb2022b7c received from {username='slave'} at 127.22.108.7:37671
I20240503 02:14:05.290736 30156 tablet_copy_service.cc:433] P a9094240860e43bfb129193f3bfd5be6: ending tablet copy session fbbc2be6382e47908ecb7c68c367bd26-5a5d5658fe5e4104a15b404cb2022b7c on tablet 5a5d5658fe5e4104a15b404cb2022b7c with peer fbbc2be6382e47908ecb7c68c367bd26
I20240503 02:14:05.301784 30005 consensus_queue.cc:237] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:14:05.304147 30264 raft_consensus.cc:1261] T 0007679ffe8047d79dd8ac778fb6f1ce P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:14:05.304291 30132 raft_consensus.cc:1261] T 0007679ffe8047d79dd8ac778fb6f1ce P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:14:05.304322 29940 consensus_peers.cc:485] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0007679ffe8047d79dd8ac778fb6f1ce. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:05.304764 10022 consensus_queue.cc:1035] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:14:05.305017 9887 consensus_queue.cc:1035] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:14:05.322417 10022 raft_consensus.cc:2932] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:05.324628 30264 raft_consensus.cc:2932] T 0007679ffe8047d79dd8ac778fb6f1ce P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:05.333328 30132 raft_consensus.cc:2932] T 0007679ffe8047d79dd8ac778fb6f1ce P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:05.335093 29725 catalog_manager.cc:5484] T 0007679ffe8047d79dd8ac778fb6f1ce P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: config changed from index -1 to 4, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:05.339128 10084 tablet_bootstrap.cc:492] T 3f00949b4ca84a49b0f6e62bd7373586 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:05.339545 10084 tablet_bootstrap.cc:492] T 3f00949b4ca84a49b0f6e62bd7373586 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap complete.
I20240503 02:14:05.339810 10084 ts_tablet_manager.cc:1367] T 3f00949b4ca84a49b0f6e62bd7373586 P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.051s user 0.046s sys 0.003s
I20240503 02:14:05.340539 10084 raft_consensus.cc:348] T 3f00949b4ca84a49b0f6e62bd7373586 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:14:05.340788 10084 raft_consensus.cc:729] T 3f00949b4ca84a49b0f6e62bd7373586 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:14:05.341090 10084 consensus_queue.cc:260] T 3f00949b4ca84a49b0f6e62bd7373586 P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:14:05.341825 10084 ts_tablet_manager.cc:1398] T 3f00949b4ca84a49b0f6e62bd7373586 P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:14:05.343422 29895 tablet_copy_service.cc:342] P 16440e365430405fb10366ab70e2b429: Request end of tablet copy session 9a4d92bc25374fb3960fde7d10be309d-3f00949b4ca84a49b0f6e62bd7373586 received from {username='slave'} at 127.22.108.5:49385
I20240503 02:14:05.343544 29895 tablet_copy_service.cc:433] P 16440e365430405fb10366ab70e2b429: ending tablet copy session 9a4d92bc25374fb3960fde7d10be309d-3f00949b4ca84a49b0f6e62bd7373586 on tablet 3f00949b4ca84a49b0f6e62bd7373586 with peer 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:14:05.376127 10084 ts_tablet_manager.cc:902] T e80b43971dc246d9bfd5e1c603f8415c P 9a4d92bc25374fb3960fde7d10be309d: Initiating tablet copy from peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:14:05.393352 10084 tablet_copy_client.cc:308] T e80b43971dc246d9bfd5e1c603f8415c P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.3:42679
I20240503 02:14:05.394335 30156 tablet_copy_service.cc:140] P a9094240860e43bfb129193f3bfd5be6: Received BeginTabletCopySession request for tablet e80b43971dc246d9bfd5e1c603f8415c from peer 9a4d92bc25374fb3960fde7d10be309d ({username='slave'} at 127.22.108.5:33825)
I20240503 02:14:05.394481 30156 tablet_copy_service.cc:161] P a9094240860e43bfb129193f3bfd5be6: Beginning new tablet copy session on tablet e80b43971dc246d9bfd5e1c603f8415c from peer 9a4d92bc25374fb3960fde7d10be309d at {username='slave'} at 127.22.108.5:33825: session id = 9a4d92bc25374fb3960fde7d10be309d-e80b43971dc246d9bfd5e1c603f8415c
I20240503 02:14:05.397557 30156 tablet_copy_source_session.cc:205] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:05.398942 10084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e80b43971dc246d9bfd5e1c603f8415c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:05.400895 30264 consensus_queue.cc:237] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:14:05.405949 10084 tablet_copy_client.cc:783] T e80b43971dc246d9bfd5e1c603f8415c P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:05.406200 10084 tablet_copy_client.cc:647] T e80b43971dc246d9bfd5e1c603f8415c P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:05.407574 29874 raft_consensus.cc:1261] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:14:05.408257 10002 consensus_queue.cc:1035] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:14:05.408490 30199 consensus_peers.cc:485] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2bf8c34ee16d4e0fbc6817d63b2ba52e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:05.409333 30132 raft_consensus.cc:1261] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:14:05.409904 10084 tablet_copy_client.cc:523] T e80b43971dc246d9bfd5e1c603f8415c P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:05.409900 10002 consensus_queue.cc:1035] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:14:05.420070 10084 tablet_bootstrap.cc:492] T e80b43971dc246d9bfd5e1c603f8415c P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:14:05.435829 10002 raft_consensus.cc:2932] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:05.436509 29873 raft_consensus.cc:2932] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:05.446065 29726 catalog_manager.cc:5484] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 16440e365430405fb10366ab70e2b429 reported cstate change: config changed from index -1 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:05.454036 30132 raft_consensus.cc:2932] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:05.486696 10084 tablet_bootstrap.cc:492] T e80b43971dc246d9bfd5e1c603f8415c P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:05.487180 10084 tablet_bootstrap.cc:492] T e80b43971dc246d9bfd5e1c603f8415c P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap complete.
I20240503 02:14:05.487556 10084 ts_tablet_manager.cc:1367] T e80b43971dc246d9bfd5e1c603f8415c P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.068s user 0.045s sys 0.000s
I20240503 02:14:05.488353 10084 raft_consensus.cc:348] T e80b43971dc246d9bfd5e1c603f8415c P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:14:05.489455 10084 raft_consensus.cc:729] T e80b43971dc246d9bfd5e1c603f8415c P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:14:05.489773 10084 consensus_queue.cc:260] T e80b43971dc246d9bfd5e1c603f8415c P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:14:05.492743 10084 ts_tablet_manager.cc:1398] T e80b43971dc246d9bfd5e1c603f8415c P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 02:14:05.497413 30156 tablet_copy_service.cc:342] P a9094240860e43bfb129193f3bfd5be6: Request end of tablet copy session 9a4d92bc25374fb3960fde7d10be309d-e80b43971dc246d9bfd5e1c603f8415c received from {username='slave'} at 127.22.108.5:33825
I20240503 02:14:05.497531 30156 tablet_copy_service.cc:433] P a9094240860e43bfb129193f3bfd5be6: ending tablet copy session 9a4d92bc25374fb3960fde7d10be309d-e80b43971dc246d9bfd5e1c603f8415c on tablet e80b43971dc246d9bfd5e1c603f8415c with peer 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:14:05.531702 30264 raft_consensus.cc:1203] T 3c437d9d466348f2863c407666c348aa P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:14:05.552196 10141 ts_tablet_manager.cc:902] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead: Initiating tablet copy from peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:14:05.553468 10141 tablet_copy_client.cc:308] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.5:35595
I20240503 02:14:05.554778 32662 tablet_copy_service.cc:140] P 9a4d92bc25374fb3960fde7d10be309d: Received BeginTabletCopySession request for tablet eec091fa0ba947d68577d562e538ab56 from peer 8a666e798fe8477d8dd1e51b385d5ead ({username='slave'} at 127.22.108.6:48381)
I20240503 02:14:05.554931 32662 tablet_copy_service.cc:161] P 9a4d92bc25374fb3960fde7d10be309d: Beginning new tablet copy session on tablet eec091fa0ba947d68577d562e538ab56 from peer 8a666e798fe8477d8dd1e51b385d5ead at {username='slave'} at 127.22.108.6:48381: session id = 8a666e798fe8477d8dd1e51b385d5ead-eec091fa0ba947d68577d562e538ab56
I20240503 02:14:05.556947 32662 tablet_copy_source_session.cc:205] T eec091fa0ba947d68577d562e538ab56 P 9a4d92bc25374fb3960fde7d10be309d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:05.558439 10141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eec091fa0ba947d68577d562e538ab56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:05.564594 10141 tablet_copy_client.cc:783] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:05.564807 10141 tablet_copy_client.cc:647] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:05.566397 10141 tablet_copy_client.cc:523] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:05.569706 10141 tablet_bootstrap.cc:492] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:14:05.574970 10154 ts_tablet_manager.cc:902] T 6635aacafed54994965ad48b2df4f163 P 8a666e798fe8477d8dd1e51b385d5ead: Initiating tablet copy from peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:14:05.575209 10141 tablet_bootstrap.cc:492] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:05.575610 10141 tablet_bootstrap.cc:492] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap complete.
I20240503 02:14:05.575868 10141 ts_tablet_manager.cc:1367] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.005s
I20240503 02:14:05.576664 10141 raft_consensus.cc:348] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:05.576952 10141 raft_consensus.cc:729] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:14:05.577270 10141 consensus_queue.cc:260] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:05.577960 10141 ts_tablet_manager.cc:1398] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:14:05.578800 32662 tablet_copy_service.cc:342] P 9a4d92bc25374fb3960fde7d10be309d: Request end of tablet copy session 8a666e798fe8477d8dd1e51b385d5ead-eec091fa0ba947d68577d562e538ab56 received from {username='slave'} at 127.22.108.6:48381
I20240503 02:14:05.578908 32662 tablet_copy_service.cc:433] P 9a4d92bc25374fb3960fde7d10be309d: ending tablet copy session 8a666e798fe8477d8dd1e51b385d5ead-eec091fa0ba947d68577d562e538ab56 on tablet eec091fa0ba947d68577d562e538ab56 with peer 8a666e798fe8477d8dd1e51b385d5ead
I20240503 02:14:05.595456 436 raft_consensus.cc:1203] T 5a5d5658fe5e4104a15b404cb2022b7c P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:14:05.600351 10154 tablet_copy_client.cc:308] T 6635aacafed54994965ad48b2df4f163 P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.2:37873
I20240503 02:14:05.601373 30026 tablet_copy_service.cc:140] P 155735c7f8b14997ac4c770aee2a9401: Received BeginTabletCopySession request for tablet 6635aacafed54994965ad48b2df4f163 from peer 8a666e798fe8477d8dd1e51b385d5ead ({username='slave'} at 127.22.108.6:33785)
I20240503 02:14:05.601521 30026 tablet_copy_service.cc:161] P 155735c7f8b14997ac4c770aee2a9401: Beginning new tablet copy session on tablet 6635aacafed54994965ad48b2df4f163 from peer 8a666e798fe8477d8dd1e51b385d5ead at {username='slave'} at 127.22.108.6:33785: session id = 8a666e798fe8477d8dd1e51b385d5ead-6635aacafed54994965ad48b2df4f163
I20240503 02:14:05.603655 30026 tablet_copy_source_session.cc:205] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:05.605142 10154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6635aacafed54994965ad48b2df4f163. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:05.611935 10154 tablet_copy_client.cc:783] T 6635aacafed54994965ad48b2df4f163 P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:05.612339 10154 tablet_copy_client.cc:647] T 6635aacafed54994965ad48b2df4f163 P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:05.615039 10154 tablet_copy_client.cc:523] T 6635aacafed54994965ad48b2df4f163 P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:05.618758 10154 tablet_bootstrap.cc:492] T 6635aacafed54994965ad48b2df4f163 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:14:05.642773 10164 ts_tablet_manager.cc:902] T c19017f829a74cc2b6dc41a2a25a3831 P f575ce92c0224eeb80b9fae5f6561fbe: Initiating tablet copy from peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:14:05.658025 10164 tablet_copy_client.cc:308] T c19017f829a74cc2b6dc41a2a25a3831 P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.4:33935
I20240503 02:14:05.658883 30286 tablet_copy_service.cc:140] P 1f10f80f13c84e80b03eda7085a3b012: Received BeginTabletCopySession request for tablet c19017f829a74cc2b6dc41a2a25a3831 from peer f575ce92c0224eeb80b9fae5f6561fbe ({username='slave'} at 127.22.108.8:51621)
I20240503 02:14:05.659018 30286 tablet_copy_service.cc:161] P 1f10f80f13c84e80b03eda7085a3b012: Beginning new tablet copy session on tablet c19017f829a74cc2b6dc41a2a25a3831 from peer f575ce92c0224eeb80b9fae5f6561fbe at {username='slave'} at 127.22.108.8:51621: session id = f575ce92c0224eeb80b9fae5f6561fbe-c19017f829a74cc2b6dc41a2a25a3831
I20240503 02:14:05.660848 30286 tablet_copy_source_session.cc:205] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:05.662160 10164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c19017f829a74cc2b6dc41a2a25a3831. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:05.671002 10164 tablet_copy_client.cc:783] T c19017f829a74cc2b6dc41a2a25a3831 P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:05.671311 10164 tablet_copy_client.cc:647] T c19017f829a74cc2b6dc41a2a25a3831 P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:05.675544 10164 tablet_copy_client.cc:523] T c19017f829a74cc2b6dc41a2a25a3831 P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:05.677618 10154 tablet_bootstrap.cc:492] T 6635aacafed54994965ad48b2df4f163 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:05.678311 10154 tablet_bootstrap.cc:492] T 6635aacafed54994965ad48b2df4f163 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap complete.
I20240503 02:14:05.678767 10154 ts_tablet_manager.cc:1367] T 6635aacafed54994965ad48b2df4f163 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.060s user 0.050s sys 0.008s
I20240503 02:14:05.680171 10154 raft_consensus.cc:348] T 6635aacafed54994965ad48b2df4f163 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:05.680433 10154 raft_consensus.cc:729] T 6635aacafed54994965ad48b2df4f163 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:14:05.680745 10154 consensus_queue.cc:260] T 6635aacafed54994965ad48b2df4f163 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:05.682574 10154 ts_tablet_manager.cc:1398] T 6635aacafed54994965ad48b2df4f163 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:14:05.684602 30026 tablet_copy_service.cc:342] P 155735c7f8b14997ac4c770aee2a9401: Request end of tablet copy session 8a666e798fe8477d8dd1e51b385d5ead-6635aacafed54994965ad48b2df4f163 received from {username='slave'} at 127.22.108.6:33785
I20240503 02:14:05.684722 30026 tablet_copy_service.cc:433] P 155735c7f8b14997ac4c770aee2a9401: ending tablet copy session 8a666e798fe8477d8dd1e51b385d5ead-6635aacafed54994965ad48b2df4f163 on tablet 6635aacafed54994965ad48b2df4f163 with peer 8a666e798fe8477d8dd1e51b385d5ead
I20240503 02:14:05.685072 10164 tablet_bootstrap.cc:492] T c19017f829a74cc2b6dc41a2a25a3831 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:14:05.713136 32641 raft_consensus.cc:1203] T 3f00949b4ca84a49b0f6e62bd7373586 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:14:05.790673 10084 ts_tablet_manager.cc:902] T 0007679ffe8047d79dd8ac778fb6f1ce P 9a4d92bc25374fb3960fde7d10be309d: Initiating tablet copy from peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:14:05.823889 10164 tablet_bootstrap.cc:492] T c19017f829a74cc2b6dc41a2a25a3831 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:05.824363 10164 tablet_bootstrap.cc:492] T c19017f829a74cc2b6dc41a2a25a3831 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap complete.
I20240503 02:14:05.824671 10164 ts_tablet_manager.cc:1367] T c19017f829a74cc2b6dc41a2a25a3831 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.140s user 0.041s sys 0.012s
I20240503 02:14:05.825686 10164 raft_consensus.cc:348] T c19017f829a74cc2b6dc41a2a25a3831 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:14:05.825903 10164 raft_consensus.cc:729] T c19017f829a74cc2b6dc41a2a25a3831 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:14:05.826123 10164 consensus_queue.cc:260] T c19017f829a74cc2b6dc41a2a25a3831 P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:14:05.826726 10164 ts_tablet_manager.cc:1398] T c19017f829a74cc2b6dc41a2a25a3831 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:14:05.827948 30286 tablet_copy_service.cc:342] P 1f10f80f13c84e80b03eda7085a3b012: Request end of tablet copy session f575ce92c0224eeb80b9fae5f6561fbe-c19017f829a74cc2b6dc41a2a25a3831 received from {username='slave'} at 127.22.108.8:51621
I20240503 02:14:05.828058 30286 tablet_copy_service.cc:433] P 1f10f80f13c84e80b03eda7085a3b012: ending tablet copy session f575ce92c0224eeb80b9fae5f6561fbe-c19017f829a74cc2b6dc41a2a25a3831 on tablet c19017f829a74cc2b6dc41a2a25a3831 with peer f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:14:05.845428 10084 tablet_copy_client.cc:308] T 0007679ffe8047d79dd8ac778fb6f1ce P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.2:37873
I20240503 02:14:05.846334 30026 tablet_copy_service.cc:140] P 155735c7f8b14997ac4c770aee2a9401: Received BeginTabletCopySession request for tablet 0007679ffe8047d79dd8ac778fb6f1ce from peer 9a4d92bc25374fb3960fde7d10be309d ({username='slave'} at 127.22.108.5:55617)
I20240503 02:14:05.846555 30026 tablet_copy_service.cc:161] P 155735c7f8b14997ac4c770aee2a9401: Beginning new tablet copy session on tablet 0007679ffe8047d79dd8ac778fb6f1ce from peer 9a4d92bc25374fb3960fde7d10be309d at {username='slave'} at 127.22.108.5:55617: session id = 9a4d92bc25374fb3960fde7d10be309d-0007679ffe8047d79dd8ac778fb6f1ce
I20240503 02:14:05.865115 30026 tablet_copy_source_session.cc:205] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:05.868392 10084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0007679ffe8047d79dd8ac778fb6f1ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:05.899238 10084 tablet_copy_client.cc:783] T 0007679ffe8047d79dd8ac778fb6f1ce P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:05.899493 10084 tablet_copy_client.cc:647] T 0007679ffe8047d79dd8ac778fb6f1ce P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:05.903616 10164 ts_tablet_manager.cc:902] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P f575ce92c0224eeb80b9fae5f6561fbe: Initiating tablet copy from peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:14:05.905331 10084 tablet_copy_client.cc:523] T 0007679ffe8047d79dd8ac778fb6f1ce P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:05.910826 10084 tablet_bootstrap.cc:492] T 0007679ffe8047d79dd8ac778fb6f1ce P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:14:05.944859 32641 raft_consensus.cc:1203] T e80b43971dc246d9bfd5e1c603f8415c P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:14:05.946467 10164 tablet_copy_client.cc:308] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.4:33935
I20240503 02:14:05.947276 30286 tablet_copy_service.cc:140] P 1f10f80f13c84e80b03eda7085a3b012: Received BeginTabletCopySession request for tablet 2bf8c34ee16d4e0fbc6817d63b2ba52e from peer f575ce92c0224eeb80b9fae5f6561fbe ({username='slave'} at 127.22.108.8:51621)
I20240503 02:14:05.947464 30286 tablet_copy_service.cc:161] P 1f10f80f13c84e80b03eda7085a3b012: Beginning new tablet copy session on tablet 2bf8c34ee16d4e0fbc6817d63b2ba52e from peer f575ce92c0224eeb80b9fae5f6561fbe at {username='slave'} at 127.22.108.8:51621: session id = f575ce92c0224eeb80b9fae5f6561fbe-2bf8c34ee16d4e0fbc6817d63b2ba52e
I20240503 02:14:05.952807 30286 tablet_copy_source_session.cc:205] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:05.955214 10164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bf8c34ee16d4e0fbc6817d63b2ba52e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:05.967579 10164 tablet_copy_client.cc:783] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:05.967844 10164 tablet_copy_client.cc:647] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:05.991982 10164 tablet_copy_client.cc:523] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:05.995714 10164 tablet_bootstrap.cc:492] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:14:06.017622 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:06.021065 10084 tablet_bootstrap.cc:492] T 0007679ffe8047d79dd8ac778fb6f1ce P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:06.021570 10084 tablet_bootstrap.cc:492] T 0007679ffe8047d79dd8ac778fb6f1ce P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap complete.
I20240503 02:14:06.021916 10084 ts_tablet_manager.cc:1367] T 0007679ffe8047d79dd8ac778fb6f1ce P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.111s user 0.047s sys 0.000s
I20240503 02:14:06.022753 10084 raft_consensus.cc:348] T 0007679ffe8047d79dd8ac778fb6f1ce P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:14:06.023049 10084 raft_consensus.cc:729] T 0007679ffe8047d79dd8ac778fb6f1ce P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:14:06.023315 10084 consensus_queue.cc:260] T 0007679ffe8047d79dd8ac778fb6f1ce P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:14:06.025449 10084 ts_tablet_manager.cc:1398] T 0007679ffe8047d79dd8ac778fb6f1ce P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
W20240503 02:14:06.056288 32574 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.7:44241, user_credentials={real_user=slave}} blocked reactor thread for 57868.3us
I20240503 02:14:06.059134 30026 tablet_copy_service.cc:342] P 155735c7f8b14997ac4c770aee2a9401: Request end of tablet copy session 9a4d92bc25374fb3960fde7d10be309d-0007679ffe8047d79dd8ac778fb6f1ce received from {username='slave'} at 127.22.108.5:55617
I20240503 02:14:06.059269 30026 tablet_copy_service.cc:433] P 155735c7f8b14997ac4c770aee2a9401: ending tablet copy session 9a4d92bc25374fb3960fde7d10be309d-0007679ffe8047d79dd8ac778fb6f1ce on tablet 0007679ffe8047d79dd8ac778fb6f1ce with peer 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:14:06.099953 305 raft_consensus.cc:1203] T 6635aacafed54994965ad48b2df4f163 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:14:06.107540 305 raft_consensus.cc:1203] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:14:06.108556 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:06.118422 565 raft_consensus.cc:1203] T c19017f829a74cc2b6dc41a2a25a3831 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:14:06.136435 10164 tablet_bootstrap.cc:492] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:06.136948 10164 tablet_bootstrap.cc:492] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap complete.
I20240503 02:14:06.137228 10164 ts_tablet_manager.cc:1367] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.142s user 0.058s sys 0.004s
I20240503 02:14:06.138027 10164 raft_consensus.cc:348] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:14:06.138252 10164 raft_consensus.cc:729] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:14:06.138475 10164 consensus_queue.cc:260] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:14:06.140548 10164 ts_tablet_manager.cc:1398] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:14:06.142634 30286 tablet_copy_service.cc:342] P 1f10f80f13c84e80b03eda7085a3b012: Request end of tablet copy session f575ce92c0224eeb80b9fae5f6561fbe-2bf8c34ee16d4e0fbc6817d63b2ba52e received from {username='slave'} at 127.22.108.8:51621
I20240503 02:14:06.142750 30286 tablet_copy_service.cc:433] P 1f10f80f13c84e80b03eda7085a3b012: ending tablet copy session f575ce92c0224eeb80b9fae5f6561fbe-2bf8c34ee16d4e0fbc6817d63b2ba52e on tablet 2bf8c34ee16d4e0fbc6817d63b2ba52e with peer f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:14:06.176127 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:06.176797 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:06.228976 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:06.233258 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:06.269121 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:06.278437 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240503 02:14:06.313220 502 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.5:35595, user_credentials={real_user=slave}} blocked reactor thread for 62104.3us
I20240503 02:14:06.401785 32641 raft_consensus.cc:1203] T 0007679ffe8047d79dd8ac778fb6f1ce P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:14:06.495961 567 raft_consensus.cc:1203] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:14:06.678239 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:06.734438 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:06.735774 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:06.751447 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:06.808686 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:06.816038 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:06.817297 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:06.884260 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:07.011296 30264 consensus_queue.cc:237] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:14:07.013474 567 raft_consensus.cc:1261] T c19017f829a74cc2b6dc41a2a25a3831 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:14:07.013556 30132 raft_consensus.cc:1261] T c19017f829a74cc2b6dc41a2a25a3831 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:14:07.014173 10002 consensus_queue.cc:1035] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:14:07.014474 10070 consensus_queue.cc:1035] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:14:07.016957 10002 raft_consensus.cc:2932] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:07.018174 30132 raft_consensus.cc:2932] T c19017f829a74cc2b6dc41a2a25a3831 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:07.019975 567 raft_consensus.cc:2932] T c19017f829a74cc2b6dc41a2a25a3831 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:07.025405 29731 catalog_manager.cc:5484] T c19017f829a74cc2b6dc41a2a25a3831 P a9094240860e43bfb129193f3bfd5be6 reported cstate change: config changed from index 4 to 5, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:07.055706 29853 tablet_service.cc:1514] Processing DeleteTablet for tablet c19017f829a74cc2b6dc41a2a25a3831 with delete_type TABLET_DATA_TOMBSTONED (TS 16440e365430405fb10366ab70e2b429 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:58744
I20240503 02:14:07.068958 10315 tablet_replica.cc:330] T c19017f829a74cc2b6dc41a2a25a3831 P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:14:07.070773 10315 raft_consensus.cc:2229] T c19017f829a74cc2b6dc41a2a25a3831 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:07.071085 10315 raft_consensus.cc:2258] T c19017f829a74cc2b6dc41a2a25a3831 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:07.071722 10315 ts_tablet_manager.cc:1875] T c19017f829a74cc2b6dc41a2a25a3831 P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:07.087997 10315 ts_tablet_manager.cc:1888] T c19017f829a74cc2b6dc41a2a25a3831 P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:14:07.088245 10315 log.cc:1196] T c19017f829a74cc2b6dc41a2a25a3831 P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/c19017f829a74cc2b6dc41a2a25a3831
I20240503 02:14:07.089010 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet c19017f829a74cc2b6dc41a2a25a3831 (table rebalance_test_table_11 [id=9498d0be60bf4564a9127c17a357f7ed]) successfully deleted
I20240503 02:14:07.412623 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:07.425590 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:07.454154 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:07.479513 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:07.513613 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:07.532902 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:07.550371 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:07.556542 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:07.705941 32638 consensus_queue.cc:237] T eec091fa0ba947d68577d562e538ab56 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:07.708038 307 raft_consensus.cc:1261] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:07.708511 567 raft_consensus.cc:1261] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:07.708581 10367 consensus_queue.cc:1035] T eec091fa0ba947d68577d562e538ab56 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:07.709203 10367 consensus_queue.cc:1035] T eec091fa0ba947d68577d562e538ab56 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:07.718385 9862 raft_consensus.cc:2932] T eec091fa0ba947d68577d562e538ab56 P 9a4d92bc25374fb3960fde7d10be309d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:07.723755 307 raft_consensus.cc:2932] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:07.724884 29729 catalog_manager.cc:5484] T eec091fa0ba947d68577d562e538ab56 P 9a4d92bc25374fb3960fde7d10be309d reported cstate change: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New cstate: current_term: 1 leader_uuid: "9a4d92bc25374fb3960fde7d10be309d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:07.734061 565 raft_consensus.cc:2932] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:07.743538 415 tablet_service.cc:1514] Processing DeleteTablet for tablet eec091fa0ba947d68577d562e538ab56 with delete_type TABLET_DATA_TOMBSTONED (TS fbbc2be6382e47908ecb7c68c367bd26 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50814
I20240503 02:14:07.744227 10413 tablet_replica.cc:330] T eec091fa0ba947d68577d562e538ab56 P fbbc2be6382e47908ecb7c68c367bd26: stopping tablet replica
I20240503 02:14:07.744427 10413 raft_consensus.cc:2229] T eec091fa0ba947d68577d562e538ab56 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:07.744750 10413 raft_consensus.cc:2258] T eec091fa0ba947d68577d562e538ab56 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:07.745519 10413 ts_tablet_manager.cc:1875] T eec091fa0ba947d68577d562e538ab56 P fbbc2be6382e47908ecb7c68c367bd26: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:07.752339 10413 ts_tablet_manager.cc:1888] T eec091fa0ba947d68577d562e538ab56 P fbbc2be6382e47908ecb7c68c367bd26: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:14:07.752447 10413 log.cc:1196] T eec091fa0ba947d68577d562e538ab56 P fbbc2be6382e47908ecb7c68c367bd26: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal/wals/eec091fa0ba947d68577d562e538ab56
I20240503 02:14:07.753185 29719 catalog_manager.cc:4830] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): tablet eec091fa0ba947d68577d562e538ab56 (table rebalancer_extra_table_42 [id=b579d0acbdab49ba82f3db319ec150ce]) successfully deleted
I20240503 02:14:08.059284 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:08.118527 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:08.123541 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:08.144526 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:08.177390 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:08.213322 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:08.288317 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:08.303104 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:08.429857 30004 consensus_queue.cc:237] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:08.431703 30265 raft_consensus.cc:1261] T 6635aacafed54994965ad48b2df4f163 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:14:08.431840 307 raft_consensus.cc:1261] T 6635aacafed54994965ad48b2df4f163 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:14:08.432228 10275 consensus_queue.cc:1035] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:14:08.432509 10276 consensus_queue.cc:1035] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:14:08.436971 10491 raft_consensus.cc:2932] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:08.437659 307 raft_consensus.cc:2932] T 6635aacafed54994965ad48b2df4f163 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:08.437445 30265 raft_consensus.cc:2932] T 6635aacafed54994965ad48b2df4f163 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:08.444684 29732 catalog_manager.cc:5484] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:08.465600 30114 tablet_service.cc:1514] Processing DeleteTablet for tablet 6635aacafed54994965ad48b2df4f163 with delete_type TABLET_DATA_TOMBSTONED (TS a9094240860e43bfb129193f3bfd5be6 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44812
I20240503 02:14:08.466655 10498 tablet_replica.cc:330] T 6635aacafed54994965ad48b2df4f163 P a9094240860e43bfb129193f3bfd5be6: stopping tablet replica
I20240503 02:14:08.466871 10498 raft_consensus.cc:2229] T 6635aacafed54994965ad48b2df4f163 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:08.467067 10498 raft_consensus.cc:2258] T 6635aacafed54994965ad48b2df4f163 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:08.467846 10498 ts_tablet_manager.cc:1875] T 6635aacafed54994965ad48b2df4f163 P a9094240860e43bfb129193f3bfd5be6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:08.473400 10498 ts_tablet_manager.cc:1888] T 6635aacafed54994965ad48b2df4f163 P a9094240860e43bfb129193f3bfd5be6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:14:08.473522 10498 log.cc:1196] T 6635aacafed54994965ad48b2df4f163 P a9094240860e43bfb129193f3bfd5be6: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/wals/6635aacafed54994965ad48b2df4f163
I20240503 02:14:08.474437 29718 catalog_manager.cc:4830] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): tablet 6635aacafed54994965ad48b2df4f163 (table rebalance_test_table_15 [id=6bbf46811ff140d289e600493149f118]) successfully deleted
I20240503 02:14:08.743569 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:08.762121 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:08.789120 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:08.798921 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:08.807858 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240503 02:14:08.849989 29809 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.6:39975, user_credentials={real_user=slave}} blocked reactor thread for 55789.1us
I20240503 02:14:08.865744 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:08.900689 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:08.925364 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:09.102933 30135 consensus_queue.cc:237] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:14:09.104795 32638 raft_consensus.cc:1261] T e80b43971dc246d9bfd5e1c603f8415c P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:14:09.104945 29873 raft_consensus.cc:1261] T e80b43971dc246d9bfd5e1c603f8415c P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:14:09.105417 10340 consensus_queue.cc:1035] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:14:09.105686 8477 consensus_queue.cc:1035] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:14:09.108412 10340 raft_consensus.cc:2932] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:09.112859 29873 raft_consensus.cc:2932] T e80b43971dc246d9bfd5e1c603f8415c P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:09.115577 29733 catalog_manager.cc:5484] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 reported cstate change: config changed from index 4 to 5, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) evicted. New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:09.122848 32638 raft_consensus.cc:2932] T e80b43971dc246d9bfd5e1c603f8415c P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:09.125691 30244 tablet_service.cc:1514] Processing DeleteTablet for tablet e80b43971dc246d9bfd5e1c603f8415c with delete_type TABLET_DATA_TOMBSTONED (TS 1f10f80f13c84e80b03eda7085a3b012 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:49856
I20240503 02:14:09.129565 10583 tablet_replica.cc:330] T e80b43971dc246d9bfd5e1c603f8415c P 1f10f80f13c84e80b03eda7085a3b012: stopping tablet replica
I20240503 02:14:09.129776 10583 raft_consensus.cc:2229] T e80b43971dc246d9bfd5e1c603f8415c P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:09.129972 10583 raft_consensus.cc:2258] T e80b43971dc246d9bfd5e1c603f8415c P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:09.130818 10583 ts_tablet_manager.cc:1875] T e80b43971dc246d9bfd5e1c603f8415c P 1f10f80f13c84e80b03eda7085a3b012: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:09.137262 10583 ts_tablet_manager.cc:1888] T e80b43971dc246d9bfd5e1c603f8415c P 1f10f80f13c84e80b03eda7085a3b012: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:14:09.137396 10583 log.cc:1196] T e80b43971dc246d9bfd5e1c603f8415c P 1f10f80f13c84e80b03eda7085a3b012: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/wals/e80b43971dc246d9bfd5e1c603f8415c
I20240503 02:14:09.138378 29719 catalog_manager.cc:4830] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): tablet e80b43971dc246d9bfd5e1c603f8415c (table rebalance_test_table_1 [id=28d7eeaed8114076b7afb44e4f89a2ea]) successfully deleted
I20240503 02:14:09.438827 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:09.439851 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:09.457872 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:09.466439 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:09.493849 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:09.522732 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:09.547477 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:09.549688 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:09.702186 30135 consensus_queue.cc:237] T 5a5d5658fe5e4104a15b404cb2022b7c P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:14:09.704650 30004 raft_consensus.cc:1261] T 5a5d5658fe5e4104a15b404cb2022b7c P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:09.705753 10340 consensus_queue.cc:1035] T 5a5d5658fe5e4104a15b404cb2022b7c P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:09.706279 435 raft_consensus.cc:1261] T 5a5d5658fe5e4104a15b404cb2022b7c P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:09.706846 10340 consensus_queue.cc:1035] T 5a5d5658fe5e4104a15b404cb2022b7c P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:09.708741 8477 raft_consensus.cc:2932] T 5a5d5658fe5e4104a15b404cb2022b7c P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:14:09.713286 30004 raft_consensus.cc:2932] T 5a5d5658fe5e4104a15b404cb2022b7c P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:14:09.714033 435 raft_consensus.cc:2932] T 5a5d5658fe5e4104a15b404cb2022b7c P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:14:09.721971 29732 catalog_manager.cc:5484] T 5a5d5658fe5e4104a15b404cb2022b7c P a9094240860e43bfb129193f3bfd5be6 reported cstate change: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:14:09.738516 547 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a5d5658fe5e4104a15b404cb2022b7c with delete_type TABLET_DATA_TOMBSTONED (TS f575ce92c0224eeb80b9fae5f6561fbe not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:39028
I20240503 02:14:09.739228 10677 tablet_replica.cc:330] T 5a5d5658fe5e4104a15b404cb2022b7c P f575ce92c0224eeb80b9fae5f6561fbe: stopping tablet replica
I20240503 02:14:09.739439 10677 raft_consensus.cc:2229] T 5a5d5658fe5e4104a15b404cb2022b7c P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:09.739658 10677 raft_consensus.cc:2258] T 5a5d5658fe5e4104a15b404cb2022b7c P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:09.740330 10677 ts_tablet_manager.cc:1875] T 5a5d5658fe5e4104a15b404cb2022b7c P f575ce92c0224eeb80b9fae5f6561fbe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:09.748114 10677 ts_tablet_manager.cc:1888] T 5a5d5658fe5e4104a15b404cb2022b7c P f575ce92c0224eeb80b9fae5f6561fbe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:14:09.748253 10677 log.cc:1196] T 5a5d5658fe5e4104a15b404cb2022b7c P f575ce92c0224eeb80b9fae5f6561fbe: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal/wals/5a5d5658fe5e4104a15b404cb2022b7c
I20240503 02:14:09.748934 29718 catalog_manager.cc:4830] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): tablet 5a5d5658fe5e4104a15b404cb2022b7c (table rebalancer_extra_table_40 [id=2285665685314d5cae1699d9f2c791f0]) successfully deleted
I20240503 02:14:10.119359 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:10.154405 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:10.157399 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:10.203769 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:10.239528 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:10.246279 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:10.288565 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:10.295935 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:10.419723 29874 consensus_queue.cc:237] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:14:10.421919 30006 raft_consensus.cc:1261] T 3f00949b4ca84a49b0f6e62bd7373586 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:14:10.421921 32640 raft_consensus.cc:1261] T 3f00949b4ca84a49b0f6e62bd7373586 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:14:10.422456 10557 consensus_queue.cc:1035] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:14:10.422726 9808 consensus_queue.cc:1035] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:14:10.425552 9808 raft_consensus.cc:2932] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:10.425953 32638 raft_consensus.cc:2932] T 3f00949b4ca84a49b0f6e62bd7373586 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:10.426679 30006 raft_consensus.cc:2932] T 3f00949b4ca84a49b0f6e62bd7373586 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:10.432343 29725 catalog_manager.cc:5484] T 3f00949b4ca84a49b0f6e62bd7373586 P 9a4d92bc25374fb3960fde7d10be309d reported cstate change: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:10.453166 30114 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f00949b4ca84a49b0f6e62bd7373586 with delete_type TABLET_DATA_TOMBSTONED (TS a9094240860e43bfb129193f3bfd5be6 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44812
I20240503 02:14:10.453831 10769 tablet_replica.cc:330] T 3f00949b4ca84a49b0f6e62bd7373586 P a9094240860e43bfb129193f3bfd5be6: stopping tablet replica
I20240503 02:14:10.454080 10769 raft_consensus.cc:2229] T 3f00949b4ca84a49b0f6e62bd7373586 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:10.454375 10769 raft_consensus.cc:2258] T 3f00949b4ca84a49b0f6e62bd7373586 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:10.455296 10769 ts_tablet_manager.cc:1875] T 3f00949b4ca84a49b0f6e62bd7373586 P a9094240860e43bfb129193f3bfd5be6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:10.461305 10769 ts_tablet_manager.cc:1888] T 3f00949b4ca84a49b0f6e62bd7373586 P a9094240860e43bfb129193f3bfd5be6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:14:10.461457 10769 log.cc:1196] T 3f00949b4ca84a49b0f6e62bd7373586 P a9094240860e43bfb129193f3bfd5be6: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/wals/3f00949b4ca84a49b0f6e62bd7373586
I20240503 02:14:10.462284 29718 catalog_manager.cc:4830] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): tablet 3f00949b4ca84a49b0f6e62bd7373586 (table rebalance_test_table_17 [id=6fdcf94cbd1844948615de6c2a4d18cc]) successfully deleted
I20240503 02:14:10.804492 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:10.809434 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:10.864156 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:10.865005 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:10.905488 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:10.942919 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:11.034060 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:11.046923 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:11.244530 305 consensus_queue.cc:237] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:14:11.246665 30264 raft_consensus.cc:1261] T 3c437d9d466348f2863c407666c348aa P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:11.246788 565 raft_consensus.cc:1261] T 3c437d9d466348f2863c407666c348aa P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:11.247241 10666 consensus_queue.cc:1035] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:11.247522 10807 consensus_queue.cc:1035] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:11.250241 10666 raft_consensus.cc:2932] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:14:11.251991 30264 raft_consensus.cc:2932] T 3c437d9d466348f2863c407666c348aa P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:14:11.253823 565 raft_consensus.cc:2932] T 3c437d9d466348f2863c407666c348aa P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:14:11.257009 29726 catalog_manager.cc:5484] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:14:11.272823 415 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c437d9d466348f2863c407666c348aa with delete_type TABLET_DATA_TOMBSTONED (TS fbbc2be6382e47908ecb7c68c367bd26 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50814
I20240503 02:14:11.273526 10851 tablet_replica.cc:330] T 3c437d9d466348f2863c407666c348aa P fbbc2be6382e47908ecb7c68c367bd26: stopping tablet replica
I20240503 02:14:11.273730 10851 raft_consensus.cc:2229] T 3c437d9d466348f2863c407666c348aa P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:11.273952 10851 raft_consensus.cc:2258] T 3c437d9d466348f2863c407666c348aa P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:11.274717 10851 ts_tablet_manager.cc:1875] T 3c437d9d466348f2863c407666c348aa P fbbc2be6382e47908ecb7c68c367bd26: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:11.280664 10851 ts_tablet_manager.cc:1888] T 3c437d9d466348f2863c407666c348aa P fbbc2be6382e47908ecb7c68c367bd26: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:14:11.280778 10851 log.cc:1196] T 3c437d9d466348f2863c407666c348aa P fbbc2be6382e47908ecb7c68c367bd26: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal/wals/3c437d9d466348f2863c407666c348aa
I20240503 02:14:11.281510 29719 catalog_manager.cc:4830] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): tablet 3c437d9d466348f2863c407666c348aa (table rebalancer_extra_table_10 [id=81455612cf1243e09bae574807b9303b]) successfully deleted
I20240503 02:14:11.593688 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:11.593822 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:11.662313 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:11.662325 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:11.663046 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:11.705442 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:11.823846 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:11.830986 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:11.981129 32640 consensus_queue.cc:237] T 28f3f05f0a344cd683cb9e010273f908 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:14:11.983323 29873 raft_consensus.cc:1261] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:11.983681 435 raft_consensus.cc:1261] T 28f3f05f0a344cd683cb9e010273f908 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:11.984215 10609 consensus_queue.cc:1035] T 28f3f05f0a344cd683cb9e010273f908 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:11.984462 10826 consensus_queue.cc:1035] T 28f3f05f0a344cd683cb9e010273f908 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:11.987155 10826 raft_consensus.cc:2932] T 28f3f05f0a344cd683cb9e010273f908 P 9a4d92bc25374fb3960fde7d10be309d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:11.988111 29873 raft_consensus.cc:2932] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:11.990176 435 raft_consensus.cc:2932] T 28f3f05f0a344cd683cb9e010273f908 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:11.996111 29734 catalog_manager.cc:5484] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 reported cstate change: config changed from index 2 to 3, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New cstate: current_term: 1 leader_uuid: "9a4d92bc25374fb3960fde7d10be309d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:12.025262 30114 tablet_service.cc:1514] Processing DeleteTablet for tablet 28f3f05f0a344cd683cb9e010273f908 with delete_type TABLET_DATA_TOMBSTONED (TS a9094240860e43bfb129193f3bfd5be6 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44812
I20240503 02:14:12.025985 10933 tablet_replica.cc:330] T 28f3f05f0a344cd683cb9e010273f908 P a9094240860e43bfb129193f3bfd5be6: stopping tablet replica
I20240503 02:14:12.026222 10933 raft_consensus.cc:2229] T 28f3f05f0a344cd683cb9e010273f908 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:12.026441 10933 raft_consensus.cc:2258] T 28f3f05f0a344cd683cb9e010273f908 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:12.027208 10933 ts_tablet_manager.cc:1875] T 28f3f05f0a344cd683cb9e010273f908 P a9094240860e43bfb129193f3bfd5be6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:12.034132 10933 ts_tablet_manager.cc:1888] T 28f3f05f0a344cd683cb9e010273f908 P a9094240860e43bfb129193f3bfd5be6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:14:12.034262 10933 log.cc:1196] T 28f3f05f0a344cd683cb9e010273f908 P a9094240860e43bfb129193f3bfd5be6: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/wals/28f3f05f0a344cd683cb9e010273f908
I20240503 02:14:12.035037 29718 catalog_manager.cc:4830] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): tablet 28f3f05f0a344cd683cb9e010273f908 (table rebalancer_extra_table_14 [id=098275b181bb4441990148dfd09ded67]) successfully deleted
I20240503 02:14:12.395126 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:12.417764 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:12.438987 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:12.467713 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:12.482751 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:12.484704 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240503 02:14:12.570535 29809 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.2:37873, user_credentials={real_user=slave}} blocked reactor thread for 159933us
I20240503 02:14:12.635270 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:12.647547 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:12.755859 30264 consensus_queue.cc:237] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:14:12.758479 29874 raft_consensus.cc:1261] T 5a44b34b585b47528c22fc678289b01a P 16440e365430405fb10366ab70e2b429 [term 2 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:14:12.759013 10835 consensus_queue.cc:1035] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:14:12.759224 566 raft_consensus.cc:1261] T 5a44b34b585b47528c22fc678289b01a P f575ce92c0224eeb80b9fae5f6561fbe [term 2 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:14:12.761652 10649 raft_consensus.cc:2932] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:12.762140 10835 consensus_queue.cc:1035] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:14:12.762977 29874 raft_consensus.cc:2932] T 5a44b34b585b47528c22fc678289b01a P 16440e365430405fb10366ab70e2b429 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:12.768046 566 raft_consensus.cc:2932] T 5a44b34b585b47528c22fc678289b01a P f575ce92c0224eeb80b9fae5f6561fbe [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:12.769304 29730 catalog_manager.cc:5484] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: config changed from index 2 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New cstate: current_term: 2 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:12.781646 32755 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a44b34b585b47528c22fc678289b01a with delete_type TABLET_DATA_TOMBSTONED (TS 8a666e798fe8477d8dd1e51b385d5ead not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46360
I20240503 02:14:12.791674 11030 tablet_replica.cc:330] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead: stopping tablet replica
I20240503 02:14:12.791890 11030 raft_consensus.cc:2229] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:12.792055 11030 raft_consensus.cc:2258] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:12.792649 11030 ts_tablet_manager.cc:1875] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:12.798300 11030 ts_tablet_manager.cc:1888] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240503 02:14:12.798425 11030 log.cc:1196] T 5a44b34b585b47528c22fc678289b01a P 8a666e798fe8477d8dd1e51b385d5ead: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal/wals/5a44b34b585b47528c22fc678289b01a
I20240503 02:14:12.799238 29719 catalog_manager.cc:4830] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): tablet 5a44b34b585b47528c22fc678289b01a (table rebalancer_extra_table_38 [id=e4b9d13872ad4d27841df71e38d5482f]) successfully deleted
I20240503 02:14:13.030965 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:13.071815 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:13.081315 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:13.144239 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:13.167791 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:13.184953 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:13.204545 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:13.253681 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:13.408460 30006 consensus_queue.cc:237] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:14:13.410463 32641 raft_consensus.cc:1261] T 0007679ffe8047d79dd8ac778fb6f1ce P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:14:13.410656 30133 raft_consensus.cc:1261] T 0007679ffe8047d79dd8ac778fb6f1ce P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:14:13.411039 10276 consensus_queue.cc:1035] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:14:13.411264 11088 consensus_queue.cc:1035] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:14:13.413941 11088 raft_consensus.cc:2932] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:13.414714 32640 raft_consensus.cc:2932] T 0007679ffe8047d79dd8ac778fb6f1ce P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:13.415354 30133 raft_consensus.cc:2932] T 0007679ffe8047d79dd8ac778fb6f1ce P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:13.420753 29729 catalog_manager.cc:5484] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: config changed from index 4 to 5, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) evicted. New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:13.435201 30244 tablet_service.cc:1514] Processing DeleteTablet for tablet 0007679ffe8047d79dd8ac778fb6f1ce with delete_type TABLET_DATA_TOMBSTONED (TS 1f10f80f13c84e80b03eda7085a3b012 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:49856
I20240503 02:14:13.435871 11117 tablet_replica.cc:330] T 0007679ffe8047d79dd8ac778fb6f1ce P 1f10f80f13c84e80b03eda7085a3b012: stopping tablet replica
I20240503 02:14:13.436165 11117 raft_consensus.cc:2229] T 0007679ffe8047d79dd8ac778fb6f1ce P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:13.436872 11117 raft_consensus.cc:2258] T 0007679ffe8047d79dd8ac778fb6f1ce P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:13.437693 11117 ts_tablet_manager.cc:1875] T 0007679ffe8047d79dd8ac778fb6f1ce P 1f10f80f13c84e80b03eda7085a3b012: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:13.444795 11117 ts_tablet_manager.cc:1888] T 0007679ffe8047d79dd8ac778fb6f1ce P 1f10f80f13c84e80b03eda7085a3b012: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:14:13.444981 11117 log.cc:1196] T 0007679ffe8047d79dd8ac778fb6f1ce P 1f10f80f13c84e80b03eda7085a3b012: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/wals/0007679ffe8047d79dd8ac778fb6f1ce
I20240503 02:14:13.449213 29719 catalog_manager.cc:4830] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): tablet 0007679ffe8047d79dd8ac778fb6f1ce (table rebalance_test_table_7 [id=f6554714a9d84d2d864dca86f3ad3642]) successfully deleted
I20240503 02:14:13.526583 32640 consensus_queue.cc:237] T 9b3b9d83e4c742bfa62624676cbb6a11 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:13.528843 435 raft_consensus.cc:1261] T 9b3b9d83e4c742bfa62624676cbb6a11 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:14:13.529100 32574 consensus_peers.cc:485] T 9b3b9d83e4c742bfa62624676cbb6a11 P 9a4d92bc25374fb3960fde7d10be309d -> Peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): Couldn't send request to peer 8a666e798fe8477d8dd1e51b385d5ead. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b3b9d83e4c742bfa62624676cbb6a11. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:13.529474 30133 raft_consensus.cc:1261] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:13.529453 10826 consensus_queue.cc:1035] T 9b3b9d83e4c742bfa62624676cbb6a11 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:13.530022 10826 consensus_queue.cc:1035] T 9b3b9d83e4c742bfa62624676cbb6a11 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:13.532521 11102 raft_consensus.cc:2932] T 9b3b9d83e4c742bfa62624676cbb6a11 P 9a4d92bc25374fb3960fde7d10be309d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:13.533136 30133 raft_consensus.cc:2932] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:13.535807 435 raft_consensus.cc:2932] T 9b3b9d83e4c742bfa62624676cbb6a11 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:13.538506 29731 catalog_manager.cc:5484] T 9b3b9d83e4c742bfa62624676cbb6a11 P 9a4d92bc25374fb3960fde7d10be309d reported cstate change: config changed from index -1 to 2, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New cstate: current_term: 1 leader_uuid: "9a4d92bc25374fb3960fde7d10be309d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:13.583884 567 consensus_queue.cc:237] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:14:13.586794 30265 raft_consensus.cc:1261] T ba97049157a646c780fa203d34cd4c82 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:14:13.586876 502 consensus_peers.cc:485] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba97049157a646c780fa203d34cd4c82. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:13.587615 10740 consensus_queue.cc:1035] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:13.588558 435 raft_consensus.cc:1261] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:13.589207 11162 consensus_queue.cc:1035] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:13.594178 11162 raft_consensus.cc:2932] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:13.594800 30265 raft_consensus.cc:2932] T ba97049157a646c780fa203d34cd4c82 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:13.601904 29729 catalog_manager.cc:5484] T ba97049157a646c780fa203d34cd4c82 P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: config changed from index -1 to 2, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New cstate: current_term: 1 leader_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:13.603617 435 raft_consensus.cc:2932] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:13.694608 435 consensus_queue.cc:237] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:14:13.696930 29874 raft_consensus.cc:1261] T f82e505ac13a41019e422b9dc1e4fef9 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:13.696939 307 raft_consensus.cc:1261] T f82e505ac13a41019e422b9dc1e4fef9 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:13.697464 11181 consensus_queue.cc:1035] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:13.697717 11081 consensus_queue.cc:1035] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:14:13.697952 370 consensus_peers.cc:485] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f82e505ac13a41019e422b9dc1e4fef9. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:13.700577 11081 raft_consensus.cc:2932] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:13.701207 307 raft_consensus.cc:2932] T f82e505ac13a41019e422b9dc1e4fef9 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:13.705518 29874 raft_consensus.cc:2932] T f82e505ac13a41019e422b9dc1e4fef9 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:13.706539 29733 catalog_manager.cc:5484] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: config changed from index -1 to 2, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:13.744939 30265 consensus_queue.cc:237] T 683d4041910f4355910e188c0b7df6c7 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:13.747632 30133 raft_consensus.cc:1261] T 683d4041910f4355910e188c0b7df6c7 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:14:13.747622 30004 raft_consensus.cc:1261] T 683d4041910f4355910e188c0b7df6c7 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:14:13.748129 11202 consensus_queue.cc:1035] T 683d4041910f4355910e188c0b7df6c7 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:14:13.748467 10649 consensus_queue.cc:1035] T 683d4041910f4355910e188c0b7df6c7 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:14:13.748713 30200 consensus_peers.cc:485] T 683d4041910f4355910e188c0b7df6c7 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): Couldn't send request to peer 8a666e798fe8477d8dd1e51b385d5ead. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 683d4041910f4355910e188c0b7df6c7. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:13.751346 11202 raft_consensus.cc:2932] T 683d4041910f4355910e188c0b7df6c7 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:13.751782 30004 raft_consensus.cc:2932] T 683d4041910f4355910e188c0b7df6c7 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:13.756214 30133 raft_consensus.cc:2932] T 683d4041910f4355910e188c0b7df6c7 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:13.758702 29731 catalog_manager.cc:5484] T 683d4041910f4355910e188c0b7df6c7 P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: config changed from index -1 to 4, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:13.792613 435 consensus_queue.cc:237] T 3351f8900c7446179da1fbe65f835e0a P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:14:13.794905 32640 raft_consensus.cc:1261] T 3351f8900c7446179da1fbe65f835e0a P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:14:13.795348 373 consensus_peers.cc:485] T 3351f8900c7446179da1fbe65f835e0a P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Couldn't send request to peer 16440e365430405fb10366ab70e2b429. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3351f8900c7446179da1fbe65f835e0a. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:13.795372 307 raft_consensus.cc:1261] T 3351f8900c7446179da1fbe65f835e0a P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:13.795606 11184 consensus_queue.cc:1035] T 3351f8900c7446179da1fbe65f835e0a P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:13.795914 11181 consensus_queue.cc:1035] T 3351f8900c7446179da1fbe65f835e0a P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:13.800366 11081 raft_consensus.cc:2932] T 3351f8900c7446179da1fbe65f835e0a P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:13.801059 305 raft_consensus.cc:2932] T 3351f8900c7446179da1fbe65f835e0a P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:13.805545 32641 raft_consensus.cc:2932] T 3351f8900c7446179da1fbe65f835e0a P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:13.807013 29733 catalog_manager.cc:5484] T 3351f8900c7446179da1fbe65f835e0a P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:14.092283 11274 ts_tablet_manager.cc:902] T 9b3b9d83e4c742bfa62624676cbb6a11 P 8a666e798fe8477d8dd1e51b385d5ead: Initiating tablet copy from peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:14:14.093758 11274 tablet_copy_client.cc:308] T 9b3b9d83e4c742bfa62624676cbb6a11 P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.5:35595
I20240503 02:14:14.094343 32662 tablet_copy_service.cc:140] P 9a4d92bc25374fb3960fde7d10be309d: Received BeginTabletCopySession request for tablet 9b3b9d83e4c742bfa62624676cbb6a11 from peer 8a666e798fe8477d8dd1e51b385d5ead ({username='slave'} at 127.22.108.6:48381)
I20240503 02:14:14.094489 32662 tablet_copy_service.cc:161] P 9a4d92bc25374fb3960fde7d10be309d: Beginning new tablet copy session on tablet 9b3b9d83e4c742bfa62624676cbb6a11 from peer 8a666e798fe8477d8dd1e51b385d5ead at {username='slave'} at 127.22.108.6:48381: session id = 8a666e798fe8477d8dd1e51b385d5ead-9b3b9d83e4c742bfa62624676cbb6a11
I20240503 02:14:14.097178 32662 tablet_copy_source_session.cc:205] T 9b3b9d83e4c742bfa62624676cbb6a11 P 9a4d92bc25374fb3960fde7d10be309d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:14.098886 11274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b3b9d83e4c742bfa62624676cbb6a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:14.106060 11274 tablet_copy_client.cc:783] T 9b3b9d83e4c742bfa62624676cbb6a11 P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:14.106269 11274 tablet_copy_client.cc:647] T 9b3b9d83e4c742bfa62624676cbb6a11 P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:14.108403 11274 tablet_copy_client.cc:523] T 9b3b9d83e4c742bfa62624676cbb6a11 P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:14.111481 11274 tablet_bootstrap.cc:492] T 9b3b9d83e4c742bfa62624676cbb6a11 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:14:14.121273 11274 tablet_bootstrap.cc:492] T 9b3b9d83e4c742bfa62624676cbb6a11 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:14.121673 11274 tablet_bootstrap.cc:492] T 9b3b9d83e4c742bfa62624676cbb6a11 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap complete.
I20240503 02:14:14.121912 11274 ts_tablet_manager.cc:1367] T 9b3b9d83e4c742bfa62624676cbb6a11 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20240503 02:14:14.122579 11274 raft_consensus.cc:348] T 9b3b9d83e4c742bfa62624676cbb6a11 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:14.122828 11274 raft_consensus.cc:729] T 9b3b9d83e4c742bfa62624676cbb6a11 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:14:14.123096 11274 consensus_queue.cc:260] T 9b3b9d83e4c742bfa62624676cbb6a11 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:14.123817 11274 ts_tablet_manager.cc:1398] T 9b3b9d83e4c742bfa62624676cbb6a11 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:14:14.124634 32662 tablet_copy_service.cc:342] P 9a4d92bc25374fb3960fde7d10be309d: Request end of tablet copy session 8a666e798fe8477d8dd1e51b385d5ead-9b3b9d83e4c742bfa62624676cbb6a11 received from {username='slave'} at 127.22.108.6:48381
I20240503 02:14:14.124733 32662 tablet_copy_service.cc:433] P 9a4d92bc25374fb3960fde7d10be309d: ending tablet copy session 8a666e798fe8477d8dd1e51b385d5ead-9b3b9d83e4c742bfa62624676cbb6a11 on tablet 9b3b9d83e4c742bfa62624676cbb6a11 with peer 8a666e798fe8477d8dd1e51b385d5ead
I20240503 02:14:14.163225 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:14.191504 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:14.198558 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:14.199306 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:14.201385 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:14.243985 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:14.265393 11296 ts_tablet_manager.cc:902] T f82e505ac13a41019e422b9dc1e4fef9 P f575ce92c0224eeb80b9fae5f6561fbe: Initiating tablet copy from peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:14:14.266691 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:14.277132 11274 ts_tablet_manager.cc:902] T 683d4041910f4355910e188c0b7df6c7 P 8a666e798fe8477d8dd1e51b385d5ead: Initiating tablet copy from peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:14:14.299888 11296 tablet_copy_client.cc:308] T f82e505ac13a41019e422b9dc1e4fef9 P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.7:44241
I20240503 02:14:14.300752 457 tablet_copy_service.cc:140] P fbbc2be6382e47908ecb7c68c367bd26: Received BeginTabletCopySession request for tablet f82e505ac13a41019e422b9dc1e4fef9 from peer f575ce92c0224eeb80b9fae5f6561fbe ({username='slave'} at 127.22.108.8:48093)
I20240503 02:14:14.300911 457 tablet_copy_service.cc:161] P fbbc2be6382e47908ecb7c68c367bd26: Beginning new tablet copy session on tablet f82e505ac13a41019e422b9dc1e4fef9 from peer f575ce92c0224eeb80b9fae5f6561fbe at {username='slave'} at 127.22.108.8:48093: session id = f575ce92c0224eeb80b9fae5f6561fbe-f82e505ac13a41019e422b9dc1e4fef9
I20240503 02:14:14.303735 457 tablet_copy_source_session.cc:205] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:14.306110 11296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f82e505ac13a41019e422b9dc1e4fef9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:14.313058 11296 tablet_copy_client.cc:783] T f82e505ac13a41019e422b9dc1e4fef9 P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:14.313393 11296 tablet_copy_client.cc:647] T f82e505ac13a41019e422b9dc1e4fef9 P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:14.313654 11274 tablet_copy_client.cc:308] T 683d4041910f4355910e188c0b7df6c7 P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.4:33935
I20240503 02:14:14.314149 11293 ts_tablet_manager.cc:902] T ba97049157a646c780fa203d34cd4c82 P 9a4d92bc25374fb3960fde7d10be309d: Initiating tablet copy from peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:14:14.314386 30286 tablet_copy_service.cc:140] P 1f10f80f13c84e80b03eda7085a3b012: Received BeginTabletCopySession request for tablet 683d4041910f4355910e188c0b7df6c7 from peer 8a666e798fe8477d8dd1e51b385d5ead ({username='slave'} at 127.22.108.6:54483)
I20240503 02:14:14.314545 30286 tablet_copy_service.cc:161] P 1f10f80f13c84e80b03eda7085a3b012: Beginning new tablet copy session on tablet 683d4041910f4355910e188c0b7df6c7 from peer 8a666e798fe8477d8dd1e51b385d5ead at {username='slave'} at 127.22.108.6:54483: session id = 8a666e798fe8477d8dd1e51b385d5ead-683d4041910f4355910e188c0b7df6c7
I20240503 02:14:14.317168 30286 tablet_copy_source_session.cc:205] T 683d4041910f4355910e188c0b7df6c7 P 1f10f80f13c84e80b03eda7085a3b012: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:14.320011 11296 tablet_copy_client.cc:523] T f82e505ac13a41019e422b9dc1e4fef9 P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:14.320641 11274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 683d4041910f4355910e188c0b7df6c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:14.323626 11296 tablet_bootstrap.cc:492] T f82e505ac13a41019e422b9dc1e4fef9 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:14:14.329722 11307 ts_tablet_manager.cc:902] T 3351f8900c7446179da1fbe65f835e0a P 16440e365430405fb10366ab70e2b429: Initiating tablet copy from peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:14:14.331399 11296 tablet_bootstrap.cc:492] T f82e505ac13a41019e422b9dc1e4fef9 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:14.332149 11293 tablet_copy_client.cc:308] T ba97049157a646c780fa203d34cd4c82 P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.8:42233
I20240503 02:14:14.332185 11296 tablet_bootstrap.cc:492] T f82e505ac13a41019e422b9dc1e4fef9 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap complete.
I20240503 02:14:14.332772 588 tablet_copy_service.cc:140] P f575ce92c0224eeb80b9fae5f6561fbe: Received BeginTabletCopySession request for tablet ba97049157a646c780fa203d34cd4c82 from peer 9a4d92bc25374fb3960fde7d10be309d ({username='slave'} at 127.22.108.5:33527)
I20240503 02:14:14.332935 588 tablet_copy_service.cc:161] P f575ce92c0224eeb80b9fae5f6561fbe: Beginning new tablet copy session on tablet ba97049157a646c780fa203d34cd4c82 from peer 9a4d92bc25374fb3960fde7d10be309d at {username='slave'} at 127.22.108.5:33527: session id = 9a4d92bc25374fb3960fde7d10be309d-ba97049157a646c780fa203d34cd4c82
I20240503 02:14:14.335302 588 tablet_copy_source_session.cc:205] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:14.332407 11296 ts_tablet_manager.cc:1367] T f82e505ac13a41019e422b9dc1e4fef9 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20240503 02:14:14.336560 11293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba97049157a646c780fa203d34cd4c82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:14.338565 11296 raft_consensus.cc:348] T f82e505ac13a41019e422b9dc1e4fef9 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:14:14.338742 11296 raft_consensus.cc:729] T f82e505ac13a41019e422b9dc1e4fef9 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:14:14.338949 11296 consensus_queue.cc:260] T f82e505ac13a41019e422b9dc1e4fef9 P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:14:14.339519 11296 ts_tablet_manager.cc:1398] T f82e505ac13a41019e422b9dc1e4fef9 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:14:14.340693 457 tablet_copy_service.cc:342] P fbbc2be6382e47908ecb7c68c367bd26: Request end of tablet copy session f575ce92c0224eeb80b9fae5f6561fbe-f82e505ac13a41019e422b9dc1e4fef9 received from {username='slave'} at 127.22.108.8:48093
I20240503 02:14:14.340847 457 tablet_copy_service.cc:433] P fbbc2be6382e47908ecb7c68c367bd26: ending tablet copy session f575ce92c0224eeb80b9fae5f6561fbe-f82e505ac13a41019e422b9dc1e4fef9 on tablet f82e505ac13a41019e422b9dc1e4fef9 with peer f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:14:14.355281 11274 tablet_copy_client.cc:783] T 683d4041910f4355910e188c0b7df6c7 P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:14.355556 11274 tablet_copy_client.cc:647] T 683d4041910f4355910e188c0b7df6c7 P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:14.357738 11274 tablet_copy_client.cc:523] T 683d4041910f4355910e188c0b7df6c7 P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:14.361065 11274 tablet_bootstrap.cc:492] T 683d4041910f4355910e188c0b7df6c7 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:14:14.365581 11293 tablet_copy_client.cc:783] T ba97049157a646c780fa203d34cd4c82 P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:14.365811 11293 tablet_copy_client.cc:647] T ba97049157a646c780fa203d34cd4c82 P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:14.378561 11307 tablet_copy_client.cc:308] T 3351f8900c7446179da1fbe65f835e0a P 16440e365430405fb10366ab70e2b429: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.7:44241
I20240503 02:14:14.381083 457 tablet_copy_service.cc:140] P fbbc2be6382e47908ecb7c68c367bd26: Received BeginTabletCopySession request for tablet 3351f8900c7446179da1fbe65f835e0a from peer 16440e365430405fb10366ab70e2b429 ({username='slave'} at 127.22.108.1:55517)
I20240503 02:14:14.381320 457 tablet_copy_service.cc:161] P fbbc2be6382e47908ecb7c68c367bd26: Beginning new tablet copy session on tablet 3351f8900c7446179da1fbe65f835e0a from peer 16440e365430405fb10366ab70e2b429 at {username='slave'} at 127.22.108.1:55517: session id = 16440e365430405fb10366ab70e2b429-3351f8900c7446179da1fbe65f835e0a
I20240503 02:14:14.401870 457 tablet_copy_source_session.cc:205] T 3351f8900c7446179da1fbe65f835e0a P fbbc2be6382e47908ecb7c68c367bd26: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:14.407817 11293 tablet_copy_client.cc:523] T ba97049157a646c780fa203d34cd4c82 P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:14.408911 11307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3351f8900c7446179da1fbe65f835e0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:14.422456 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:14.423373 11293 tablet_bootstrap.cc:492] T ba97049157a646c780fa203d34cd4c82 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:14:14.459048 11307 tablet_copy_client.cc:783] T 3351f8900c7446179da1fbe65f835e0a P 16440e365430405fb10366ab70e2b429: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:14.459565 11307 tablet_copy_client.cc:647] T 3351f8900c7446179da1fbe65f835e0a P 16440e365430405fb10366ab70e2b429: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:14.465317 11307 tablet_copy_client.cc:523] T 3351f8900c7446179da1fbe65f835e0a P 16440e365430405fb10366ab70e2b429: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:14.471370 11293 tablet_bootstrap.cc:492] T ba97049157a646c780fa203d34cd4c82 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:14.471755 11293 tablet_bootstrap.cc:492] T ba97049157a646c780fa203d34cd4c82 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap complete.
I20240503 02:14:14.471992 11293 ts_tablet_manager.cc:1367] T ba97049157a646c780fa203d34cd4c82 P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.049s user 0.004s sys 0.016s
I20240503 02:14:14.472713 11293 raft_consensus.cc:348] T ba97049157a646c780fa203d34cd4c82 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:14:14.472927 11293 raft_consensus.cc:729] T ba97049157a646c780fa203d34cd4c82 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:14:14.473649 11293 consensus_queue.cc:260] T ba97049157a646c780fa203d34cd4c82 P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:14:14.475956 11293 ts_tablet_manager.cc:1398] T ba97049157a646c780fa203d34cd4c82 P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 02:14:14.476006 11307 tablet_bootstrap.cc:492] T 3351f8900c7446179da1fbe65f835e0a P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:14:14.482090 588 tablet_copy_service.cc:342] P f575ce92c0224eeb80b9fae5f6561fbe: Request end of tablet copy session 9a4d92bc25374fb3960fde7d10be309d-ba97049157a646c780fa203d34cd4c82 received from {username='slave'} at 127.22.108.5:33527
I20240503 02:14:14.482214 588 tablet_copy_service.cc:433] P f575ce92c0224eeb80b9fae5f6561fbe: ending tablet copy session 9a4d92bc25374fb3960fde7d10be309d-ba97049157a646c780fa203d34cd4c82 on tablet ba97049157a646c780fa203d34cd4c82 with peer 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:14:14.491462 305 raft_consensus.cc:1203] T 9b3b9d83e4c742bfa62624676cbb6a11 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:14:14.491695 11307 tablet_bootstrap.cc:492] T 3351f8900c7446179da1fbe65f835e0a P 16440e365430405fb10366ab70e2b429: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:14.492067 11307 tablet_bootstrap.cc:492] T 3351f8900c7446179da1fbe65f835e0a P 16440e365430405fb10366ab70e2b429: Bootstrap complete.
I20240503 02:14:14.492372 11307 ts_tablet_manager.cc:1367] T 3351f8900c7446179da1fbe65f835e0a P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20240503 02:14:14.493310 11307 raft_consensus.cc:348] T 3351f8900c7446179da1fbe65f835e0a P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:14:14.493515 11307 raft_consensus.cc:729] T 3351f8900c7446179da1fbe65f835e0a P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:14:14.493716 11307 consensus_queue.cc:260] T 3351f8900c7446179da1fbe65f835e0a P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:14:14.494654 11307 ts_tablet_manager.cc:1398] T 3351f8900c7446179da1fbe65f835e0a P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:14:14.496147 457 tablet_copy_service.cc:342] P fbbc2be6382e47908ecb7c68c367bd26: Request end of tablet copy session 16440e365430405fb10366ab70e2b429-3351f8900c7446179da1fbe65f835e0a received from {username='slave'} at 127.22.108.1:55517
I20240503 02:14:14.496264 457 tablet_copy_service.cc:433] P fbbc2be6382e47908ecb7c68c367bd26: ending tablet copy session 16440e365430405fb10366ab70e2b429-3351f8900c7446179da1fbe65f835e0a on tablet 3351f8900c7446179da1fbe65f835e0a with peer 16440e365430405fb10366ab70e2b429
I20240503 02:14:14.546294 11274 tablet_bootstrap.cc:492] T 683d4041910f4355910e188c0b7df6c7 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:14.546744 11274 tablet_bootstrap.cc:492] T 683d4041910f4355910e188c0b7df6c7 P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap complete.
I20240503 02:14:14.547029 11274 ts_tablet_manager.cc:1367] T 683d4041910f4355910e188c0b7df6c7 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.186s user 0.085s sys 0.037s
I20240503 02:14:14.547839 11274 raft_consensus.cc:348] T 683d4041910f4355910e188c0b7df6c7 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:14.548107 11274 raft_consensus.cc:729] T 683d4041910f4355910e188c0b7df6c7 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:14:14.548401 11274 consensus_queue.cc:260] T 683d4041910f4355910e188c0b7df6c7 P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:14.549129 11274 ts_tablet_manager.cc:1398] T 683d4041910f4355910e188c0b7df6c7 P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 02:14:14.550204 30286 tablet_copy_service.cc:342] P 1f10f80f13c84e80b03eda7085a3b012: Request end of tablet copy session 8a666e798fe8477d8dd1e51b385d5ead-683d4041910f4355910e188c0b7df6c7 received from {username='slave'} at 127.22.108.6:54483
I20240503 02:14:14.550316 30286 tablet_copy_service.cc:433] P 1f10f80f13c84e80b03eda7085a3b012: ending tablet copy session 8a666e798fe8477d8dd1e51b385d5ead-683d4041910f4355910e188c0b7df6c7 on tablet 683d4041910f4355910e188c0b7df6c7 with peer 8a666e798fe8477d8dd1e51b385d5ead
I20240503 02:14:14.661484 305 raft_consensus.cc:1203] T 683d4041910f4355910e188c0b7df6c7 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:14:14.786047 567 raft_consensus.cc:1203] T f82e505ac13a41019e422b9dc1e4fef9 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:14:14.794795 32641 raft_consensus.cc:1203] T ba97049157a646c780fa203d34cd4c82 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:14:14.827790 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:14.838261 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:14.843703 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:14.874645 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:14.891795 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:14.893980 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:14.901933 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:14.902801 29874 raft_consensus.cc:1203] T 3351f8900c7446179da1fbe65f835e0a P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:14:15.004673 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:15.136535 32641 consensus_queue.cc:237] T 9b3b9d83e4c742bfa62624676cbb6a11 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:15.138861 30133 raft_consensus.cc:1261] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:15.139058 307 raft_consensus.cc:1261] T 9b3b9d83e4c742bfa62624676cbb6a11 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:15.139736 10826 consensus_queue.cc:1035] T 9b3b9d83e4c742bfa62624676cbb6a11 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:15.140048 10609 consensus_queue.cc:1035] T 9b3b9d83e4c742bfa62624676cbb6a11 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:15.143733 10826 raft_consensus.cc:2932] T 9b3b9d83e4c742bfa62624676cbb6a11 P 9a4d92bc25374fb3960fde7d10be309d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:15.144184 307 raft_consensus.cc:2932] T 9b3b9d83e4c742bfa62624676cbb6a11 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:15.149715 30133 raft_consensus.cc:2932] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:15.151036 29732 catalog_manager.cc:5484] T 9b3b9d83e4c742bfa62624676cbb6a11 P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: config changed from index 2 to 3, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) evicted. New cstate: current_term: 1 leader_uuid: "9a4d92bc25374fb3960fde7d10be309d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:15.199234 415 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b3b9d83e4c742bfa62624676cbb6a11 with delete_type TABLET_DATA_TOMBSTONED (TS fbbc2be6382e47908ecb7c68c367bd26 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50814
I20240503 02:14:15.212145 11411 tablet_replica.cc:330] T 9b3b9d83e4c742bfa62624676cbb6a11 P fbbc2be6382e47908ecb7c68c367bd26: stopping tablet replica
I20240503 02:14:15.212482 11411 raft_consensus.cc:2229] T 9b3b9d83e4c742bfa62624676cbb6a11 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:15.212759 11411 raft_consensus.cc:2258] T 9b3b9d83e4c742bfa62624676cbb6a11 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:15.213754 11411 ts_tablet_manager.cc:1875] T 9b3b9d83e4c742bfa62624676cbb6a11 P fbbc2be6382e47908ecb7c68c367bd26: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:15.223605 11411 ts_tablet_manager.cc:1888] T 9b3b9d83e4c742bfa62624676cbb6a11 P fbbc2be6382e47908ecb7c68c367bd26: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:14:15.224702 11411 log.cc:1196] T 9b3b9d83e4c742bfa62624676cbb6a11 P fbbc2be6382e47908ecb7c68c367bd26: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal/wals/9b3b9d83e4c742bfa62624676cbb6a11
I20240503 02:14:15.226853 29719 catalog_manager.cc:4830] TS fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): tablet 9b3b9d83e4c742bfa62624676cbb6a11 (table rebalancer_extra_table_48 [id=196b7559fe174e1b963b8d38f2240e48]) successfully deleted
I20240503 02:14:15.460325 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:15.504624 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:15.511189 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:15.568676 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:15.570472 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:15.589848 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:15.594132 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:15.651481 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:15.777482 30264 consensus_queue.cc:237] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:14:15.779958 565 raft_consensus.cc:1261] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:14:15.780550 11381 consensus_queue.cc:1035] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:14:15.781144 29874 raft_consensus.cc:1261] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:14:15.782370 11381 consensus_queue.cc:1035] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:14:15.785964 11381 raft_consensus.cc:2932] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:15.786342 29874 raft_consensus.cc:2932] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:15.792477 29732 catalog_manager.cc:5484] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 16440e365430405fb10366ab70e2b429 reported cstate change: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:15.792726 565 raft_consensus.cc:2932] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:15.805261 30114 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bf8c34ee16d4e0fbc6817d63b2ba52e with delete_type TABLET_DATA_TOMBSTONED (TS a9094240860e43bfb129193f3bfd5be6 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44812
I20240503 02:14:15.806398 11492 tablet_replica.cc:330] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P a9094240860e43bfb129193f3bfd5be6: stopping tablet replica
I20240503 02:14:15.806646 11492 raft_consensus.cc:2229] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:15.806847 11492 raft_consensus.cc:2258] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:15.807590 11492 ts_tablet_manager.cc:1875] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P a9094240860e43bfb129193f3bfd5be6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:15.815349 11492 ts_tablet_manager.cc:1888] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P a9094240860e43bfb129193f3bfd5be6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:14:15.815466 11492 log.cc:1196] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P a9094240860e43bfb129193f3bfd5be6: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/wals/2bf8c34ee16d4e0fbc6817d63b2ba52e
I20240503 02:14:15.816177 29718 catalog_manager.cc:4830] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): tablet 2bf8c34ee16d4e0fbc6817d63b2ba52e (table rebalance_test_table_7 [id=f6554714a9d84d2d864dca86f3ad3642]) successfully deleted
I20240503 02:14:16.120889 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:16.140604 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:16.238844 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:16.276450 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:16.320432 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:16.320691 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:16.325737 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:16.373857 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:16.496152 30264 consensus_queue.cc:237] T 683d4041910f4355910e188c0b7df6c7 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:16.499639 305 raft_consensus.cc:1261] T 683d4041910f4355910e188c0b7df6c7 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:14:16.499745 30004 raft_consensus.cc:1261] T 683d4041910f4355910e188c0b7df6c7 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:14:16.500260 11483 consensus_queue.cc:1035] T 683d4041910f4355910e188c0b7df6c7 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:14:16.500517 11381 consensus_queue.cc:1035] T 683d4041910f4355910e188c0b7df6c7 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:14:16.503252 11483 raft_consensus.cc:2932] T 683d4041910f4355910e188c0b7df6c7 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:16.505602 30004 raft_consensus.cc:2932] T 683d4041910f4355910e188c0b7df6c7 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:16.505654 305 raft_consensus.cc:2932] T 683d4041910f4355910e188c0b7df6c7 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:16.512192 29729 catalog_manager.cc:5484] T 683d4041910f4355910e188c0b7df6c7 P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:16.531931 30114 tablet_service.cc:1514] Processing DeleteTablet for tablet 683d4041910f4355910e188c0b7df6c7 with delete_type TABLET_DATA_TOMBSTONED (TS a9094240860e43bfb129193f3bfd5be6 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44812
I20240503 02:14:16.532541 11591 tablet_replica.cc:330] T 683d4041910f4355910e188c0b7df6c7 P a9094240860e43bfb129193f3bfd5be6: stopping tablet replica
I20240503 02:14:16.532799 11591 raft_consensus.cc:2229] T 683d4041910f4355910e188c0b7df6c7 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:16.533094 11591 raft_consensus.cc:2258] T 683d4041910f4355910e188c0b7df6c7 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:16.533869 11591 ts_tablet_manager.cc:1875] T 683d4041910f4355910e188c0b7df6c7 P a9094240860e43bfb129193f3bfd5be6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:16.540652 11591 ts_tablet_manager.cc:1888] T 683d4041910f4355910e188c0b7df6c7 P a9094240860e43bfb129193f3bfd5be6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:14:16.540840 11591 log.cc:1196] T 683d4041910f4355910e188c0b7df6c7 P a9094240860e43bfb129193f3bfd5be6: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/wals/683d4041910f4355910e188c0b7df6c7
I20240503 02:14:16.541793 29718 catalog_manager.cc:4830] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): tablet 683d4041910f4355910e188c0b7df6c7 (table rebalance_test_table_3 [id=3131069dba4a4f9aa26d4c134fcb4704]) successfully deleted
I20240503 02:14:16.849277 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:16.863941 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:16.881309 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:16.882380 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:16.925364 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:16.928598 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:16.941776 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:16.960067 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:17.015250 29723 maintenance_manager.cc:419] P e72f3141a78a4d0e811377c9c1f11f00: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033381
W20240503 02:14:17.034667 29942 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.1:36719, user_credentials={real_user=slave}} blocked reactor thread for 86159.6us
I20240503 02:14:17.149659 29722 maintenance_manager.cc:643] P e72f3141a78a4d0e811377c9c1f11f00: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.134s user 0.066s sys 0.017s Metrics: {"bytes_written":91528,"cfile_init":1,"dirs.queue_time_us":21467,"dirs.run_cpu_time_us":259,"dirs.run_wall_time_us":926,"drs_written":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1612,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":253,"thread_start_us":22092,"threads_started":1}
I20240503 02:14:17.151152 29723 maintenance_manager.cc:419] P e72f3141a78a4d0e811377c9c1f11f00: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 30590 bytes on disk
I20240503 02:14:17.152077 29722 maintenance_manager.cc:643] P e72f3141a78a4d0e811377c9c1f11f00: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20240503 02:14:17.194097 435 consensus_queue.cc:237] T 3351f8900c7446179da1fbe65f835e0a P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:14:17.196036 29874 raft_consensus.cc:1261] T 3351f8900c7446179da1fbe65f835e0a P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:17.196139 307 raft_consensus.cc:1261] T 3351f8900c7446179da1fbe65f835e0a P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:17.196614 11081 consensus_queue.cc:1035] T 3351f8900c7446179da1fbe65f835e0a P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:17.196899 11181 consensus_queue.cc:1035] T 3351f8900c7446179da1fbe65f835e0a P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:17.199590 11081 raft_consensus.cc:2932] T 3351f8900c7446179da1fbe65f835e0a P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:17.200009 307 raft_consensus.cc:2932] T 3351f8900c7446179da1fbe65f835e0a P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:17.201979 29874 raft_consensus.cc:2932] T 3351f8900c7446179da1fbe65f835e0a P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:17.207300 29729 catalog_manager.cc:5484] T 3351f8900c7446179da1fbe65f835e0a P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:17.231396 32619 tablet_service.cc:1514] Processing DeleteTablet for tablet 3351f8900c7446179da1fbe65f835e0a with delete_type TABLET_DATA_TOMBSTONED (TS 9a4d92bc25374fb3960fde7d10be309d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:38200
I20240503 02:14:17.236961 11705 tablet_replica.cc:330] T 3351f8900c7446179da1fbe65f835e0a P 9a4d92bc25374fb3960fde7d10be309d: stopping tablet replica
I20240503 02:14:17.237298 11705 raft_consensus.cc:2229] T 3351f8900c7446179da1fbe65f835e0a P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:17.237586 11705 raft_consensus.cc:2258] T 3351f8900c7446179da1fbe65f835e0a P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:17.238477 11705 ts_tablet_manager.cc:1875] T 3351f8900c7446179da1fbe65f835e0a P 9a4d92bc25374fb3960fde7d10be309d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:17.248546 11705 ts_tablet_manager.cc:1888] T 3351f8900c7446179da1fbe65f835e0a P 9a4d92bc25374fb3960fde7d10be309d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:14:17.248781 11705 log.cc:1196] T 3351f8900c7446179da1fbe65f835e0a P 9a4d92bc25374fb3960fde7d10be309d: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal/wals/3351f8900c7446179da1fbe65f835e0a
I20240503 02:14:17.249608 29719 catalog_manager.cc:4830] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): tablet 3351f8900c7446179da1fbe65f835e0a (table rebalancer_extra_table_44 [id=d7c477fdad9640c89f1806779ac9630f]) successfully deleted
I20240503 02:14:17.565943 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:17.571334 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:17.604007 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:17.611431 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:17.619818 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:17.628535 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:17.634667 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:17.682729 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:17.813014 436 consensus_queue.cc:237] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:14:17.815089 567 raft_consensus.cc:1261] T f82e505ac13a41019e422b9dc1e4fef9 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:17.815212 305 raft_consensus.cc:1261] T f82e505ac13a41019e422b9dc1e4fef9 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:17.815631 11181 consensus_queue.cc:1035] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:17.815891 11081 consensus_queue.cc:1035] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:17.822540 11081 raft_consensus.cc:2932] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:17.823395 567 raft_consensus.cc:2932] T f82e505ac13a41019e422b9dc1e4fef9 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:17.823680 305 raft_consensus.cc:2932] T f82e505ac13a41019e422b9dc1e4fef9 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:17.831849 29732 catalog_manager.cc:5484] T f82e505ac13a41019e422b9dc1e4fef9 P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: config changed from index 2 to 3, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:17.863116 29853 tablet_service.cc:1514] Processing DeleteTablet for tablet f82e505ac13a41019e422b9dc1e4fef9 with delete_type TABLET_DATA_TOMBSTONED (TS 16440e365430405fb10366ab70e2b429 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:58744
I20240503 02:14:17.865417 11799 tablet_replica.cc:330] T f82e505ac13a41019e422b9dc1e4fef9 P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:14:17.865710 11799 raft_consensus.cc:2229] T f82e505ac13a41019e422b9dc1e4fef9 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:17.865958 11799 raft_consensus.cc:2258] T f82e505ac13a41019e422b9dc1e4fef9 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:17.866570 11799 ts_tablet_manager.cc:1875] T f82e505ac13a41019e422b9dc1e4fef9 P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:17.876078 11799 ts_tablet_manager.cc:1888] T f82e505ac13a41019e422b9dc1e4fef9 P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:14:17.876528 11799 log.cc:1196] T f82e505ac13a41019e422b9dc1e4fef9 P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/f82e505ac13a41019e422b9dc1e4fef9
I20240503 02:14:17.877421 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet f82e505ac13a41019e422b9dc1e4fef9 (table rebalancer_extra_table_8 [id=c14bbf6b6bfd41148c82b54717aceb15]) successfully deleted
I20240503 02:14:17.939350 567 consensus_queue.cc:237] T eb0f4fe6b8684f329725ac2f7d5428f2 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:14:17.941793 29874 raft_consensus.cc:1261] T eb0f4fe6b8684f329725ac2f7d5428f2 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:17.942075 305 raft_consensus.cc:1261] T eb0f4fe6b8684f329725ac2f7d5428f2 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:14:17.942699 502 consensus_peers.cc:485] T eb0f4fe6b8684f329725ac2f7d5428f2 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): Couldn't send request to peer fbbc2be6382e47908ecb7c68c367bd26. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb0f4fe6b8684f329725ac2f7d5428f2. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:17.942839 11162 consensus_queue.cc:1035] T eb0f4fe6b8684f329725ac2f7d5428f2 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:17.943109 11817 consensus_queue.cc:1035] T eb0f4fe6b8684f329725ac2f7d5428f2 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:17.958276 11162 raft_consensus.cc:2932] T eb0f4fe6b8684f329725ac2f7d5428f2 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:14:17.958880 29874 raft_consensus.cc:2932] T eb0f4fe6b8684f329725ac2f7d5428f2 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:14:17.965226 305 raft_consensus.cc:2932] T eb0f4fe6b8684f329725ac2f7d5428f2 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:14:17.966061 29726 catalog_manager.cc:5484] T eb0f4fe6b8684f329725ac2f7d5428f2 P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: config changed from index -1 to 2, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New cstate: current_term: 1 leader_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:14:18.010458 32641 consensus_queue.cc:237] T e8a76caac93d441b92e65380bf6dc03e P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:18.013410 436 raft_consensus.cc:1261] T e8a76caac93d441b92e65380bf6dc03e P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:18.014146 11401 consensus_queue.cc:1035] T e8a76caac93d441b92e65380bf6dc03e P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:14:18.014935 32573 consensus_peers.cc:485] T e8a76caac93d441b92e65380bf6dc03e P 9a4d92bc25374fb3960fde7d10be309d -> Peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Couldn't send request to peer a9094240860e43bfb129193f3bfd5be6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e8a76caac93d441b92e65380bf6dc03e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:18.015332 567 raft_consensus.cc:1261] T e8a76caac93d441b92e65380bf6dc03e P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:18.015901 11401 consensus_queue.cc:1035] T e8a76caac93d441b92e65380bf6dc03e P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:18.035516 11401 raft_consensus.cc:2932] T e8a76caac93d441b92e65380bf6dc03e P 9a4d92bc25374fb3960fde7d10be309d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:18.036175 567 raft_consensus.cc:2932] T e8a76caac93d441b92e65380bf6dc03e P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:18.039276 435 raft_consensus.cc:2932] T e8a76caac93d441b92e65380bf6dc03e P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:18.047088 29725 catalog_manager.cc:5484] T e8a76caac93d441b92e65380bf6dc03e P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New cstate: current_term: 1 leader_uuid: "9a4d92bc25374fb3960fde7d10be309d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:18.084908 435 consensus_queue.cc:237] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
W20240503 02:14:18.088860 373 consensus_peers.cc:485] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Couldn't send request to peer 16440e365430405fb10366ab70e2b429. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 08e4d119d90f4146ba53a1b18efe2386. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:18.089047 305 raft_consensus.cc:1261] T 08e4d119d90f4146ba53a1b18efe2386 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:18.089572 11181 consensus_queue.cc:1035] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:18.090020 32641 raft_consensus.cc:1261] T 08e4d119d90f4146ba53a1b18efe2386 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:18.091140 11181 consensus_queue.cc:1035] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:18.098414 11181 raft_consensus.cc:2932] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:18.098912 32641 raft_consensus.cc:2932] T 08e4d119d90f4146ba53a1b18efe2386 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:18.102517 305 raft_consensus.cc:2932] T 08e4d119d90f4146ba53a1b18efe2386 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:18.106611 29725 catalog_manager.cc:5484] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:18.170444 567 consensus_queue.cc:237] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:18.173110 307 raft_consensus.cc:1261] T 9be5395489a3488da62f4d5b66facf3d P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:18.173322 32641 raft_consensus.cc:1261] T 9be5395489a3488da62f4d5b66facf3d P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:14:18.173663 501 consensus_peers.cc:485] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe -> Peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Couldn't send request to peer a9094240860e43bfb129193f3bfd5be6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9be5395489a3488da62f4d5b66facf3d. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:18.173754 11820 consensus_queue.cc:1035] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:18.174181 11162 consensus_queue.cc:1035] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:18.201557 11162 raft_consensus.cc:2932] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:18.207087 305 raft_consensus.cc:2932] T 9be5395489a3488da62f4d5b66facf3d P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:18.209151 32640 raft_consensus.cc:2932] T 9be5395489a3488da62f4d5b66facf3d P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:18.210436 29725 catalog_manager.cc:5484] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New cstate: current_term: 1 leader_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:18.267233 30265 consensus_queue.cc:237] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:14:18.269960 29874 raft_consensus.cc:1261] T fa288463ae154e28afd9c71b4b55ef3a P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:14:18.270298 30135 raft_consensus.cc:1261] T fa288463ae154e28afd9c71b4b55ef3a P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:14:18.270763 11483 consensus_queue.cc:1035] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:14:18.272704 11483 consensus_queue.cc:1035] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:14:18.273653 30199 consensus_peers.cc:485] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fa288463ae154e28afd9c71b4b55ef3a. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:18.286754 11483 raft_consensus.cc:2932] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:18.287546 29874 raft_consensus.cc:2932] T fa288463ae154e28afd9c71b4b55ef3a P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:18.290757 30135 raft_consensus.cc:2932] T fa288463ae154e28afd9c71b4b55ef3a P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:18.300475 29726 catalog_manager.cc:5484] T fa288463ae154e28afd9c71b4b55ef3a P 16440e365430405fb10366ab70e2b429 reported cstate change: config changed from index -1 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:18.329368 435 consensus_queue.cc:237] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:14:18.331861 32640 raft_consensus.cc:1261] T 62d5e9ea7af8435bb34028df2b3f1457 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:18.332288 567 raft_consensus.cc:1261] T 62d5e9ea7af8435bb34028df2b3f1457 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:18.332367 11795 consensus_queue.cc:1035] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:14:18.332764 373 consensus_peers.cc:485] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Couldn't send request to peer 16440e365430405fb10366ab70e2b429. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62d5e9ea7af8435bb34028df2b3f1457. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:18.332962 11795 consensus_queue.cc:1035] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:18.335590 11081 raft_consensus.cc:2932] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:18.336210 567 raft_consensus.cc:2932] T 62d5e9ea7af8435bb34028df2b3f1457 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:18.340292 32640 raft_consensus.cc:2932] T 62d5e9ea7af8435bb34028df2b3f1457 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:18.344875 29725 catalog_manager.cc:5484] T 62d5e9ea7af8435bb34028df2b3f1457 P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:18.409731 435 consensus_queue.cc:237] T 88f696d03330448d879ed7b60364c28c P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:14:18.411901 30265 raft_consensus.cc:1261] T 88f696d03330448d879ed7b60364c28c P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:18.412218 32640 raft_consensus.cc:1261] T 88f696d03330448d879ed7b60364c28c P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:18.412487 11081 consensus_queue.cc:1035] T 88f696d03330448d879ed7b60364c28c P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:14:18.412833 373 consensus_peers.cc:485] T 88f696d03330448d879ed7b60364c28c P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Couldn't send request to peer 16440e365430405fb10366ab70e2b429. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 88f696d03330448d879ed7b60364c28c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:18.413493 11795 consensus_queue.cc:1035] T 88f696d03330448d879ed7b60364c28c P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:18.422132 11081 raft_consensus.cc:2932] T 88f696d03330448d879ed7b60364c28c P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:18.422748 30265 raft_consensus.cc:2932] T 88f696d03330448d879ed7b60364c28c P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:18.424430 11934 ts_tablet_manager.cc:902] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26: Initiating tablet copy from peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:14:18.425968 11934 tablet_copy_client.cc:308] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.8:42233
I20240503 02:14:18.426545 588 tablet_copy_service.cc:140] P f575ce92c0224eeb80b9fae5f6561fbe: Received BeginTabletCopySession request for tablet eb0f4fe6b8684f329725ac2f7d5428f2 from peer fbbc2be6382e47908ecb7c68c367bd26 ({username='slave'} at 127.22.108.7:53877)
I20240503 02:14:18.426728 588 tablet_copy_service.cc:161] P f575ce92c0224eeb80b9fae5f6561fbe: Beginning new tablet copy session on tablet eb0f4fe6b8684f329725ac2f7d5428f2 from peer fbbc2be6382e47908ecb7c68c367bd26 at {username='slave'} at 127.22.108.7:53877: session id = fbbc2be6382e47908ecb7c68c367bd26-eb0f4fe6b8684f329725ac2f7d5428f2
I20240503 02:14:18.429016 32640 raft_consensus.cc:2932] T 88f696d03330448d879ed7b60364c28c P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:18.432569 588 tablet_copy_source_session.cc:205] T eb0f4fe6b8684f329725ac2f7d5428f2 P f575ce92c0224eeb80b9fae5f6561fbe: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:18.433916 11934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb0f4fe6b8684f329725ac2f7d5428f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:18.434168 29732 catalog_manager.cc:5484] T 88f696d03330448d879ed7b60364c28c P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:18.444595 11934 tablet_copy_client.cc:783] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:18.444898 11934 tablet_copy_client.cc:647] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:18.446883 11934 tablet_copy_client.cc:523] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:18.449968 11934 tablet_bootstrap.cc:492] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:14:18.458207 11934 tablet_bootstrap.cc:492] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:18.458603 11934 tablet_bootstrap.cc:492] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap complete.
I20240503 02:14:18.458860 11934 ts_tablet_manager.cc:1367] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20240503 02:14:18.459575 11934 raft_consensus.cc:348] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:14:18.459861 11934 raft_consensus.cc:729] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:14:18.460091 11934 consensus_queue.cc:260] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:14:18.460717 11934 ts_tablet_manager.cc:1398] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:14:18.467841 588 tablet_copy_service.cc:342] P f575ce92c0224eeb80b9fae5f6561fbe: Request end of tablet copy session fbbc2be6382e47908ecb7c68c367bd26-eb0f4fe6b8684f329725ac2f7d5428f2 received from {username='slave'} at 127.22.108.7:53877
I20240503 02:14:18.468006 588 tablet_copy_service.cc:433] P f575ce92c0224eeb80b9fae5f6561fbe: ending tablet copy session fbbc2be6382e47908ecb7c68c367bd26-eb0f4fe6b8684f329725ac2f7d5428f2 on tablet eb0f4fe6b8684f329725ac2f7d5428f2 with peer fbbc2be6382e47908ecb7c68c367bd26
I20240503 02:14:18.685767 11990 ts_tablet_manager.cc:902] T 08e4d119d90f4146ba53a1b18efe2386 P 16440e365430405fb10366ab70e2b429: Initiating tablet copy from peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:14:18.701889 11990 tablet_copy_client.cc:308] T 08e4d119d90f4146ba53a1b18efe2386 P 16440e365430405fb10366ab70e2b429: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.7:44241
I20240503 02:14:18.702862 11983 ts_tablet_manager.cc:902] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6: Initiating tablet copy from peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:14:18.709378 457 tablet_copy_service.cc:140] P fbbc2be6382e47908ecb7c68c367bd26: Received BeginTabletCopySession request for tablet 08e4d119d90f4146ba53a1b18efe2386 from peer 16440e365430405fb10366ab70e2b429 ({username='slave'} at 127.22.108.1:55517)
I20240503 02:14:18.709568 457 tablet_copy_service.cc:161] P fbbc2be6382e47908ecb7c68c367bd26: Beginning new tablet copy session on tablet 08e4d119d90f4146ba53a1b18efe2386 from peer 16440e365430405fb10366ab70e2b429 at {username='slave'} at 127.22.108.1:55517: session id = 16440e365430405fb10366ab70e2b429-08e4d119d90f4146ba53a1b18efe2386
I20240503 02:14:18.710178 11998 ts_tablet_manager.cc:902] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6: Initiating tablet copy from peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:14:18.713480 457 tablet_copy_source_session.cc:205] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:18.714327 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:18.716221 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:18.724736 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:18.729700 11990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08e4d119d90f4146ba53a1b18efe2386. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:18.797199 12004 ts_tablet_manager.cc:902] T fa288463ae154e28afd9c71b4b55ef3a P f575ce92c0224eeb80b9fae5f6561fbe: Initiating tablet copy from peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:14:18.809468 436 raft_consensus.cc:1203] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:14:18.815317 11990 tablet_copy_client.cc:783] T 08e4d119d90f4146ba53a1b18efe2386 P 16440e365430405fb10366ab70e2b429: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:18.815577 11990 tablet_copy_client.cc:647] T 08e4d119d90f4146ba53a1b18efe2386 P 16440e365430405fb10366ab70e2b429: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:18.845389 12004 tablet_copy_client.cc:308] T fa288463ae154e28afd9c71b4b55ef3a P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.4:33935
I20240503 02:14:18.846702 30286 tablet_copy_service.cc:140] P 1f10f80f13c84e80b03eda7085a3b012: Received BeginTabletCopySession request for tablet fa288463ae154e28afd9c71b4b55ef3a from peer f575ce92c0224eeb80b9fae5f6561fbe ({username='slave'} at 127.22.108.8:51621)
I20240503 02:14:18.846860 30286 tablet_copy_service.cc:161] P 1f10f80f13c84e80b03eda7085a3b012: Beginning new tablet copy session on tablet fa288463ae154e28afd9c71b4b55ef3a from peer f575ce92c0224eeb80b9fae5f6561fbe at {username='slave'} at 127.22.108.8:51621: session id = f575ce92c0224eeb80b9fae5f6561fbe-fa288463ae154e28afd9c71b4b55ef3a
I20240503 02:14:18.849551 30286 tablet_copy_source_session.cc:205] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:18.851435 11990 tablet_copy_client.cc:523] T 08e4d119d90f4146ba53a1b18efe2386 P 16440e365430405fb10366ab70e2b429: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:18.859230 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:18.861757 12004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa288463ae154e28afd9c71b4b55ef3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:18.863878 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:18.869282 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:18.884068 11998 tablet_copy_client.cc:308] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.8:42233
I20240503 02:14:18.885968 11990 tablet_bootstrap.cc:492] T 08e4d119d90f4146ba53a1b18efe2386 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:14:18.887485 588 tablet_copy_service.cc:140] P f575ce92c0224eeb80b9fae5f6561fbe: Received BeginTabletCopySession request for tablet 9be5395489a3488da62f4d5b66facf3d from peer a9094240860e43bfb129193f3bfd5be6 ({username='slave'} at 127.22.108.3:57589)
I20240503 02:14:18.887656 588 tablet_copy_service.cc:161] P f575ce92c0224eeb80b9fae5f6561fbe: Beginning new tablet copy session on tablet 9be5395489a3488da62f4d5b66facf3d from peer a9094240860e43bfb129193f3bfd5be6 at {username='slave'} at 127.22.108.3:57589: session id = a9094240860e43bfb129193f3bfd5be6-9be5395489a3488da62f4d5b66facf3d
I20240503 02:14:18.890493 588 tablet_copy_source_session.cc:205] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:18.893153 11998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9be5395489a3488da62f4d5b66facf3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:18.909940 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:18.916870 12017 ts_tablet_manager.cc:902] T 62d5e9ea7af8435bb34028df2b3f1457 P 16440e365430405fb10366ab70e2b429: Initiating tablet copy from peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:14:18.924091 12004 tablet_copy_client.cc:783] T fa288463ae154e28afd9c71b4b55ef3a P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:18.924490 12004 tablet_copy_client.cc:647] T fa288463ae154e28afd9c71b4b55ef3a P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:18.931206 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:18.940346 12004 tablet_copy_client.cc:523] T fa288463ae154e28afd9c71b4b55ef3a P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:18.944554 12004 tablet_bootstrap.cc:492] T fa288463ae154e28afd9c71b4b55ef3a P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:14:18.945479 11983 tablet_copy_client.cc:308] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.5:35595
I20240503 02:14:18.946269 32662 tablet_copy_service.cc:140] P 9a4d92bc25374fb3960fde7d10be309d: Received BeginTabletCopySession request for tablet e8a76caac93d441b92e65380bf6dc03e from peer a9094240860e43bfb129193f3bfd5be6 ({username='slave'} at 127.22.108.3:41959)
I20240503 02:14:18.946439 32662 tablet_copy_service.cc:161] P 9a4d92bc25374fb3960fde7d10be309d: Beginning new tablet copy session on tablet e8a76caac93d441b92e65380bf6dc03e from peer a9094240860e43bfb129193f3bfd5be6 at {username='slave'} at 127.22.108.3:41959: session id = a9094240860e43bfb129193f3bfd5be6-e8a76caac93d441b92e65380bf6dc03e
I20240503 02:14:18.947919 12017 tablet_copy_client.cc:308] T 62d5e9ea7af8435bb34028df2b3f1457 P 16440e365430405fb10366ab70e2b429: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.7:44241
I20240503 02:14:18.949597 457 tablet_copy_service.cc:140] P fbbc2be6382e47908ecb7c68c367bd26: Received BeginTabletCopySession request for tablet 62d5e9ea7af8435bb34028df2b3f1457 from peer 16440e365430405fb10366ab70e2b429 ({username='slave'} at 127.22.108.1:55517)
I20240503 02:14:18.949780 457 tablet_copy_service.cc:161] P fbbc2be6382e47908ecb7c68c367bd26: Beginning new tablet copy session on tablet 62d5e9ea7af8435bb34028df2b3f1457 from peer 16440e365430405fb10366ab70e2b429 at {username='slave'} at 127.22.108.1:55517: session id = 16440e365430405fb10366ab70e2b429-62d5e9ea7af8435bb34028df2b3f1457
I20240503 02:14:18.950680 32662 tablet_copy_source_session.cc:205] T e8a76caac93d441b92e65380bf6dc03e P 9a4d92bc25374fb3960fde7d10be309d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:18.952190 11983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8a76caac93d441b92e65380bf6dc03e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:18.957103 11998 tablet_copy_client.cc:783] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:18.957412 11998 tablet_copy_client.cc:647] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:18.961337 457 tablet_copy_source_session.cc:205] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:18.963028 11990 tablet_bootstrap.cc:492] T 08e4d119d90f4146ba53a1b18efe2386 P 16440e365430405fb10366ab70e2b429: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:18.963120 12017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62d5e9ea7af8435bb34028df2b3f1457. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:18.967289 11990 tablet_bootstrap.cc:492] T 08e4d119d90f4146ba53a1b18efe2386 P 16440e365430405fb10366ab70e2b429: Bootstrap complete.
I20240503 02:14:18.967641 11990 ts_tablet_manager.cc:1367] T 08e4d119d90f4146ba53a1b18efe2386 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.082s user 0.007s sys 0.000s
I20240503 02:14:18.967785 11998 tablet_copy_client.cc:523] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:18.980450 12017 tablet_copy_client.cc:783] T 62d5e9ea7af8435bb34028df2b3f1457 P 16440e365430405fb10366ab70e2b429: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:18.980693 12017 tablet_copy_client.cc:647] T 62d5e9ea7af8435bb34028df2b3f1457 P 16440e365430405fb10366ab70e2b429: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:18.981122 11990 raft_consensus.cc:348] T 08e4d119d90f4146ba53a1b18efe2386 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:14:18.981235 11983 tablet_copy_client.cc:783] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:18.981555 11983 tablet_copy_client.cc:647] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:18.983392 11983 tablet_copy_client.cc:523] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:18.983314 12017 tablet_copy_client.cc:523] T 62d5e9ea7af8435bb34028df2b3f1457 P 16440e365430405fb10366ab70e2b429: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:18.983958 11990 raft_consensus.cc:729] T 08e4d119d90f4146ba53a1b18efe2386 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:14:18.988828 11990 consensus_queue.cc:260] T 08e4d119d90f4146ba53a1b18efe2386 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:14:18.991168 12017 tablet_bootstrap.cc:492] T 62d5e9ea7af8435bb34028df2b3f1457 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:14:18.995721 11990 ts_tablet_manager.cc:1398] T 08e4d119d90f4146ba53a1b18efe2386 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 02:14:18.996598 457 tablet_copy_service.cc:342] P fbbc2be6382e47908ecb7c68c367bd26: Request end of tablet copy session 16440e365430405fb10366ab70e2b429-08e4d119d90f4146ba53a1b18efe2386 received from {username='slave'} at 127.22.108.1:55517
I20240503 02:14:18.996773 457 tablet_copy_service.cc:433] P fbbc2be6382e47908ecb7c68c367bd26: ending tablet copy session 16440e365430405fb10366ab70e2b429-08e4d119d90f4146ba53a1b18efe2386 on tablet 08e4d119d90f4146ba53a1b18efe2386 with peer 16440e365430405fb10366ab70e2b429
I20240503 02:14:19.000452 11983 tablet_bootstrap.cc:492] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:14:19.007550 11990 ts_tablet_manager.cc:902] T 88f696d03330448d879ed7b60364c28c P 16440e365430405fb10366ab70e2b429: Initiating tablet copy from peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:14:19.010203 11998 tablet_bootstrap.cc:492] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:14:19.011940 12017 tablet_bootstrap.cc:492] T 62d5e9ea7af8435bb34028df2b3f1457 P 16440e365430405fb10366ab70e2b429: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:19.012521 12017 tablet_bootstrap.cc:492] T 62d5e9ea7af8435bb34028df2b3f1457 P 16440e365430405fb10366ab70e2b429: Bootstrap complete.
I20240503 02:14:19.012874 12017 ts_tablet_manager.cc:1367] T 62d5e9ea7af8435bb34028df2b3f1457 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.003s
I20240503 02:14:19.015594 11983 tablet_bootstrap.cc:492] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:19.016058 11983 tablet_bootstrap.cc:492] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6: Bootstrap complete.
I20240503 02:14:19.016321 11983 ts_tablet_manager.cc:1367] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.001s
I20240503 02:14:19.016287 12017 raft_consensus.cc:348] T 62d5e9ea7af8435bb34028df2b3f1457 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:14:19.017187 11983 raft_consensus.cc:348] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:19.017385 11983 raft_consensus.cc:729] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:14:19.017540 12017 raft_consensus.cc:729] T 62d5e9ea7af8435bb34028df2b3f1457 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:14:19.017583 11983 consensus_queue.cc:260] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:19.018066 11983 ts_tablet_manager.cc:1398] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:14:19.018687 32662 tablet_copy_service.cc:342] P 9a4d92bc25374fb3960fde7d10be309d: Request end of tablet copy session a9094240860e43bfb129193f3bfd5be6-e8a76caac93d441b92e65380bf6dc03e received from {username='slave'} at 127.22.108.3:41959
I20240503 02:14:19.018815 32662 tablet_copy_service.cc:433] P 9a4d92bc25374fb3960fde7d10be309d: ending tablet copy session a9094240860e43bfb129193f3bfd5be6-e8a76caac93d441b92e65380bf6dc03e on tablet e8a76caac93d441b92e65380bf6dc03e with peer a9094240860e43bfb129193f3bfd5be6
I20240503 02:14:19.020401 12017 consensus_queue.cc:260] T 62d5e9ea7af8435bb34028df2b3f1457 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:14:19.022328 12017 ts_tablet_manager.cc:1398] T 62d5e9ea7af8435bb34028df2b3f1457 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 02:14:19.035338 457 tablet_copy_service.cc:342] P fbbc2be6382e47908ecb7c68c367bd26: Request end of tablet copy session 16440e365430405fb10366ab70e2b429-62d5e9ea7af8435bb34028df2b3f1457 received from {username='slave'} at 127.22.108.1:55517
I20240503 02:14:19.035496 457 tablet_copy_service.cc:433] P fbbc2be6382e47908ecb7c68c367bd26: ending tablet copy session 16440e365430405fb10366ab70e2b429-62d5e9ea7af8435bb34028df2b3f1457 on tablet 62d5e9ea7af8435bb34028df2b3f1457 with peer 16440e365430405fb10366ab70e2b429
I20240503 02:14:19.072976 11998 tablet_bootstrap.cc:492] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:19.073506 11998 tablet_bootstrap.cc:492] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6: Bootstrap complete.
I20240503 02:14:19.073798 11998 ts_tablet_manager.cc:1367] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.064s user 0.008s sys 0.000s
I20240503 02:14:19.074565 11998 raft_consensus.cc:348] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:19.074816 11998 raft_consensus.cc:729] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:14:19.075059 11998 consensus_queue.cc:260] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:19.075635 11998 ts_tablet_manager.cc:1398] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:14:19.080588 588 tablet_copy_service.cc:342] P f575ce92c0224eeb80b9fae5f6561fbe: Request end of tablet copy session a9094240860e43bfb129193f3bfd5be6-9be5395489a3488da62f4d5b66facf3d received from {username='slave'} at 127.22.108.3:57589
I20240503 02:14:19.080744 588 tablet_copy_service.cc:433] P f575ce92c0224eeb80b9fae5f6561fbe: ending tablet copy session a9094240860e43bfb129193f3bfd5be6-9be5395489a3488da62f4d5b66facf3d on tablet 9be5395489a3488da62f4d5b66facf3d with peer a9094240860e43bfb129193f3bfd5be6
I20240503 02:14:19.100175 11990 tablet_copy_client.cc:308] T 88f696d03330448d879ed7b60364c28c P 16440e365430405fb10366ab70e2b429: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.7:44241
I20240503 02:14:19.101048 457 tablet_copy_service.cc:140] P fbbc2be6382e47908ecb7c68c367bd26: Received BeginTabletCopySession request for tablet 88f696d03330448d879ed7b60364c28c from peer 16440e365430405fb10366ab70e2b429 ({username='slave'} at 127.22.108.1:55517)
I20240503 02:14:19.101235 457 tablet_copy_service.cc:161] P fbbc2be6382e47908ecb7c68c367bd26: Beginning new tablet copy session on tablet 88f696d03330448d879ed7b60364c28c from peer 16440e365430405fb10366ab70e2b429 at {username='slave'} at 127.22.108.1:55517: session id = 16440e365430405fb10366ab70e2b429-88f696d03330448d879ed7b60364c28c
I20240503 02:14:19.104254 457 tablet_copy_source_session.cc:205] T 88f696d03330448d879ed7b60364c28c P fbbc2be6382e47908ecb7c68c367bd26: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:19.105531 11990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88f696d03330448d879ed7b60364c28c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:19.112758 11990 tablet_copy_client.cc:783] T 88f696d03330448d879ed7b60364c28c P 16440e365430405fb10366ab70e2b429: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:19.113101 11990 tablet_copy_client.cc:647] T 88f696d03330448d879ed7b60364c28c P 16440e365430405fb10366ab70e2b429: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:19.118180 11990 tablet_copy_client.cc:523] T 88f696d03330448d879ed7b60364c28c P 16440e365430405fb10366ab70e2b429: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:19.121351 11990 tablet_bootstrap.cc:492] T 88f696d03330448d879ed7b60364c28c P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:14:19.128069 11990 tablet_bootstrap.cc:492] T 88f696d03330448d879ed7b60364c28c P 16440e365430405fb10366ab70e2b429: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:19.128631 11990 tablet_bootstrap.cc:492] T 88f696d03330448d879ed7b60364c28c P 16440e365430405fb10366ab70e2b429: Bootstrap complete.
I20240503 02:14:19.128988 11990 ts_tablet_manager.cc:1367] T 88f696d03330448d879ed7b60364c28c P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20240503 02:14:19.129773 11990 raft_consensus.cc:348] T 88f696d03330448d879ed7b60364c28c P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:14:19.129976 11990 raft_consensus.cc:729] T 88f696d03330448d879ed7b60364c28c P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:14:19.130242 11990 consensus_queue.cc:260] T 88f696d03330448d879ed7b60364c28c P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:14:19.132160 11990 ts_tablet_manager.cc:1398] T 88f696d03330448d879ed7b60364c28c P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:14:19.133337 457 tablet_copy_service.cc:342] P fbbc2be6382e47908ecb7c68c367bd26: Request end of tablet copy session 16440e365430405fb10366ab70e2b429-88f696d03330448d879ed7b60364c28c received from {username='slave'} at 127.22.108.1:55517
I20240503 02:14:19.133458 457 tablet_copy_service.cc:433] P fbbc2be6382e47908ecb7c68c367bd26: ending tablet copy session 16440e365430405fb10366ab70e2b429-88f696d03330448d879ed7b60364c28c on tablet 88f696d03330448d879ed7b60364c28c with peer 16440e365430405fb10366ab70e2b429
I20240503 02:14:19.142876 29873 raft_consensus.cc:1203] T 08e4d119d90f4146ba53a1b18efe2386 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:14:19.144479 12004 tablet_bootstrap.cc:492] T fa288463ae154e28afd9c71b4b55ef3a P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:19.144969 12004 tablet_bootstrap.cc:492] T fa288463ae154e28afd9c71b4b55ef3a P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap complete.
I20240503 02:14:19.145288 12004 ts_tablet_manager.cc:1367] T fa288463ae154e28afd9c71b4b55ef3a P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.201s user 0.044s sys 0.004s
I20240503 02:14:19.146135 12004 raft_consensus.cc:348] T fa288463ae154e28afd9c71b4b55ef3a P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:14:19.146364 12004 raft_consensus.cc:729] T fa288463ae154e28afd9c71b4b55ef3a P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:14:19.149000 12004 consensus_queue.cc:260] T fa288463ae154e28afd9c71b4b55ef3a P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:14:19.150969 12004 ts_tablet_manager.cc:1398] T fa288463ae154e28afd9c71b4b55ef3a P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 02:14:19.151605 30286 tablet_copy_service.cc:342] P 1f10f80f13c84e80b03eda7085a3b012: Request end of tablet copy session f575ce92c0224eeb80b9fae5f6561fbe-fa288463ae154e28afd9c71b4b55ef3a received from {username='slave'} at 127.22.108.8:51621
I20240503 02:14:19.151757 30286 tablet_copy_service.cc:433] P 1f10f80f13c84e80b03eda7085a3b012: ending tablet copy session f575ce92c0224eeb80b9fae5f6561fbe-fa288463ae154e28afd9c71b4b55ef3a on tablet fa288463ae154e28afd9c71b4b55ef3a with peer f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:14:19.181612 30133 raft_consensus.cc:1203] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:14:19.230408 30133 raft_consensus.cc:1203] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:14:19.238674 565 raft_consensus.cc:1203] T fa288463ae154e28afd9c71b4b55ef3a P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:14:19.312304 29873 raft_consensus.cc:1203] T 62d5e9ea7af8435bb34028df2b3f1457 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:14:19.506620 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:19.512427 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:19.583189 29873 raft_consensus.cc:1203] T 88f696d03330448d879ed7b60364c28c P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:14:19.594866 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:19.595561 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:19.596360 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:19.596873 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:19.617363 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:19.651284 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:19.776770 436 consensus_queue.cc:237] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:14:19.786191 29873 raft_consensus.cc:1261] T 62d5e9ea7af8435bb34028df2b3f1457 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:19.786240 567 raft_consensus.cc:1261] T 62d5e9ea7af8435bb34028df2b3f1457 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:19.786813 11939 consensus_queue.cc:1035] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:19.787225 11081 consensus_queue.cc:1035] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:19.789431 11939 raft_consensus.cc:2932] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:19.789971 29873 raft_consensus.cc:2932] T 62d5e9ea7af8435bb34028df2b3f1457 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:19.790166 565 raft_consensus.cc:2932] T 62d5e9ea7af8435bb34028df2b3f1457 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:19.797549 29727 catalog_manager.cc:5484] T 62d5e9ea7af8435bb34028df2b3f1457 P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:19.817214 32619 tablet_service.cc:1514] Processing DeleteTablet for tablet 62d5e9ea7af8435bb34028df2b3f1457 with delete_type TABLET_DATA_TOMBSTONED (TS 9a4d92bc25374fb3960fde7d10be309d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:38200
I20240503 02:14:19.817864 12126 tablet_replica.cc:330] T 62d5e9ea7af8435bb34028df2b3f1457 P 9a4d92bc25374fb3960fde7d10be309d: stopping tablet replica
I20240503 02:14:19.818163 12126 raft_consensus.cc:2229] T 62d5e9ea7af8435bb34028df2b3f1457 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:19.818490 12126 raft_consensus.cc:2258] T 62d5e9ea7af8435bb34028df2b3f1457 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:19.819150 12126 ts_tablet_manager.cc:1875] T 62d5e9ea7af8435bb34028df2b3f1457 P 9a4d92bc25374fb3960fde7d10be309d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:19.824733 12126 ts_tablet_manager.cc:1888] T 62d5e9ea7af8435bb34028df2b3f1457 P 9a4d92bc25374fb3960fde7d10be309d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:14:19.824887 12126 log.cc:1196] T 62d5e9ea7af8435bb34028df2b3f1457 P 9a4d92bc25374fb3960fde7d10be309d: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal/wals/62d5e9ea7af8435bb34028df2b3f1457
I20240503 02:14:19.825697 29719 catalog_manager.cc:4830] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): tablet 62d5e9ea7af8435bb34028df2b3f1457 (table rebalancer_extra_table_16 [id=97e0c0e7261b408f9cb42ad58ba75222]) successfully deleted
I20240503 02:14:20.194576 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:20.202268 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:20.240074 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:20.298354 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:20.326692 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:20.350412 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:20.377861 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:20.382508 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240503 02:14:20.418823 32708 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.3:42679, user_credentials={real_user=slave}} blocked reactor thread for 80421.2us
I20240503 02:14:20.528703 30264 consensus_queue.cc:237] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:14:20.531340 29873 raft_consensus.cc:1261] T fa288463ae154e28afd9c71b4b55ef3a P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:14:20.532130 11483 consensus_queue.cc:1035] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:14:20.532413 567 raft_consensus.cc:1261] T fa288463ae154e28afd9c71b4b55ef3a P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:14:20.533004 11483 consensus_queue.cc:1035] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:14:20.537000 11897 raft_consensus.cc:2932] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:20.540140 565 raft_consensus.cc:2932] T fa288463ae154e28afd9c71b4b55ef3a P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:20.540164 29873 raft_consensus.cc:2932] T fa288463ae154e28afd9c71b4b55ef3a P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:20.543035 29727 catalog_manager.cc:5484] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: config changed from index 4 to 5, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:20.555799 30114 tablet_service.cc:1514] Processing DeleteTablet for tablet fa288463ae154e28afd9c71b4b55ef3a with delete_type TABLET_DATA_TOMBSTONED (TS a9094240860e43bfb129193f3bfd5be6 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44812
I20240503 02:14:20.559149 12220 tablet_replica.cc:330] T fa288463ae154e28afd9c71b4b55ef3a P a9094240860e43bfb129193f3bfd5be6: stopping tablet replica
I20240503 02:14:20.559356 12220 raft_consensus.cc:2229] T fa288463ae154e28afd9c71b4b55ef3a P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:20.559723 12220 raft_consensus.cc:2258] T fa288463ae154e28afd9c71b4b55ef3a P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:20.560531 12220 ts_tablet_manager.cc:1875] T fa288463ae154e28afd9c71b4b55ef3a P a9094240860e43bfb129193f3bfd5be6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:20.565304 12220 ts_tablet_manager.cc:1888] T fa288463ae154e28afd9c71b4b55ef3a P a9094240860e43bfb129193f3bfd5be6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:14:20.565464 12220 log.cc:1196] T fa288463ae154e28afd9c71b4b55ef3a P a9094240860e43bfb129193f3bfd5be6: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/wals/fa288463ae154e28afd9c71b4b55ef3a
I20240503 02:14:20.566102 29718 catalog_manager.cc:4830] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): tablet fa288463ae154e28afd9c71b4b55ef3a (table rebalance_test_table_9 [id=e80de094521a4d71bc2fb5f3343275b8]) successfully deleted
I20240503 02:14:20.811125 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:20.811317 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:20.815023 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:20.815439 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:20.831305 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:20.847685 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:20.864339 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:20.894691 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:21.052489 435 consensus_queue.cc:237] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:14:21.058493 29873 raft_consensus.cc:1261] T 08e4d119d90f4146ba53a1b18efe2386 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:21.059063 305 raft_consensus.cc:1261] T 08e4d119d90f4146ba53a1b18efe2386 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:21.059387 11939 consensus_queue.cc:1035] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:21.059643 12113 consensus_queue.cc:1035] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:21.062906 11939 raft_consensus.cc:2932] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:21.065624 29873 raft_consensus.cc:2932] T 08e4d119d90f4146ba53a1b18efe2386 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:21.070852 305 raft_consensus.cc:2932] T 08e4d119d90f4146ba53a1b18efe2386 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:21.071461 29733 catalog_manager.cc:5484] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:21.091928 32619 tablet_service.cc:1514] Processing DeleteTablet for tablet 08e4d119d90f4146ba53a1b18efe2386 with delete_type TABLET_DATA_TOMBSTONED (TS 9a4d92bc25374fb3960fde7d10be309d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:38200
I20240503 02:14:21.099069 12297 tablet_replica.cc:330] T 08e4d119d90f4146ba53a1b18efe2386 P 9a4d92bc25374fb3960fde7d10be309d: stopping tablet replica
I20240503 02:14:21.099306 12297 raft_consensus.cc:2229] T 08e4d119d90f4146ba53a1b18efe2386 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:21.099521 12297 raft_consensus.cc:2258] T 08e4d119d90f4146ba53a1b18efe2386 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:21.100195 12297 ts_tablet_manager.cc:1875] T 08e4d119d90f4146ba53a1b18efe2386 P 9a4d92bc25374fb3960fde7d10be309d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:21.106024 12297 ts_tablet_manager.cc:1888] T 08e4d119d90f4146ba53a1b18efe2386 P 9a4d92bc25374fb3960fde7d10be309d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:14:21.106153 12297 log.cc:1196] T 08e4d119d90f4146ba53a1b18efe2386 P 9a4d92bc25374fb3960fde7d10be309d: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal/wals/08e4d119d90f4146ba53a1b18efe2386
I20240503 02:14:21.106909 29719 catalog_manager.cc:4830] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): tablet 08e4d119d90f4146ba53a1b18efe2386 (table rebalancer_extra_table_28 [id=d70cb2e6817a47df8925321a6a778740]) successfully deleted
I20240503 02:14:21.364966 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:21.393779 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:21.396178 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:21.405277 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:21.441313 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:21.456012 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:21.456161 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:21.485769 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:21.685791 32640 consensus_queue.cc:237] T e8a76caac93d441b92e65380bf6dc03e P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:21.687952 435 raft_consensus.cc:1261] T e8a76caac93d441b92e65380bf6dc03e P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:21.688001 30135 raft_consensus.cc:1261] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:21.688555 12185 consensus_queue.cc:1035] T e8a76caac93d441b92e65380bf6dc03e P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:21.689007 11401 consensus_queue.cc:1035] T e8a76caac93d441b92e65380bf6dc03e P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:21.698284 12185 raft_consensus.cc:2932] T e8a76caac93d441b92e65380bf6dc03e P 9a4d92bc25374fb3960fde7d10be309d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:21.703308 30135 raft_consensus.cc:2932] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:21.704424 435 raft_consensus.cc:2932] T e8a76caac93d441b92e65380bf6dc03e P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:21.710273 29730 catalog_manager.cc:5484] T e8a76caac93d441b92e65380bf6dc03e P 9a4d92bc25374fb3960fde7d10be309d reported cstate change: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New cstate: current_term: 1 leader_uuid: "9a4d92bc25374fb3960fde7d10be309d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:21.736328 547 tablet_service.cc:1514] Processing DeleteTablet for tablet e8a76caac93d441b92e65380bf6dc03e with delete_type TABLET_DATA_TOMBSTONED (TS f575ce92c0224eeb80b9fae5f6561fbe not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:39028
I20240503 02:14:21.739367 12381 tablet_replica.cc:330] T e8a76caac93d441b92e65380bf6dc03e P f575ce92c0224eeb80b9fae5f6561fbe: stopping tablet replica
I20240503 02:14:21.739676 12381 raft_consensus.cc:2229] T e8a76caac93d441b92e65380bf6dc03e P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:21.740039 12381 raft_consensus.cc:2258] T e8a76caac93d441b92e65380bf6dc03e P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:21.740792 12381 ts_tablet_manager.cc:1875] T e8a76caac93d441b92e65380bf6dc03e P f575ce92c0224eeb80b9fae5f6561fbe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:21.747319 12381 ts_tablet_manager.cc:1888] T e8a76caac93d441b92e65380bf6dc03e P f575ce92c0224eeb80b9fae5f6561fbe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:14:21.747470 12381 log.cc:1196] T e8a76caac93d441b92e65380bf6dc03e P f575ce92c0224eeb80b9fae5f6561fbe: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal/wals/e8a76caac93d441b92e65380bf6dc03e
I20240503 02:14:21.748201 29718 catalog_manager.cc:4830] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): tablet e8a76caac93d441b92e65380bf6dc03e (table rebalancer_extra_table_24 [id=782c508495e74448b6859bfc4ff35de0]) successfully deleted
I20240503 02:14:22.140868 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:22.160254 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:22.171888 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:22.227105 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:22.242451 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:22.254848 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:22.272663 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:22.318608 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:22.441004 565 consensus_queue.cc:237] T eb0f4fe6b8684f329725ac2f7d5428f2 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:14:22.442961 29873 raft_consensus.cc:1261] T eb0f4fe6b8684f329725ac2f7d5428f2 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:22.442961 434 raft_consensus.cc:1261] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:22.443490 12352 consensus_queue.cc:1035] T eb0f4fe6b8684f329725ac2f7d5428f2 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:22.443768 12351 consensus_queue.cc:1035] T eb0f4fe6b8684f329725ac2f7d5428f2 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:22.447011 12351 raft_consensus.cc:2932] T eb0f4fe6b8684f329725ac2f7d5428f2 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:14:22.447444 29873 raft_consensus.cc:2932] T eb0f4fe6b8684f329725ac2f7d5428f2 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:14:22.455183 29732 catalog_manager.cc:5484] T eb0f4fe6b8684f329725ac2f7d5428f2 P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: config changed from index 2 to 3, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New cstate: current_term: 1 leader_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:14:22.460779 434 raft_consensus.cc:2932] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:14:22.468806 32755 tablet_service.cc:1514] Processing DeleteTablet for tablet eb0f4fe6b8684f329725ac2f7d5428f2 with delete_type TABLET_DATA_TOMBSTONED (TS 8a666e798fe8477d8dd1e51b385d5ead not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46360
I20240503 02:14:22.469403 12469 tablet_replica.cc:330] T eb0f4fe6b8684f329725ac2f7d5428f2 P 8a666e798fe8477d8dd1e51b385d5ead: stopping tablet replica
I20240503 02:14:22.469656 12469 raft_consensus.cc:2229] T eb0f4fe6b8684f329725ac2f7d5428f2 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:22.469882 12469 raft_consensus.cc:2258] T eb0f4fe6b8684f329725ac2f7d5428f2 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:22.470593 12469 ts_tablet_manager.cc:1875] T eb0f4fe6b8684f329725ac2f7d5428f2 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:22.477869 12469 ts_tablet_manager.cc:1888] T eb0f4fe6b8684f329725ac2f7d5428f2 P 8a666e798fe8477d8dd1e51b385d5ead: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:14:22.478017 12469 log.cc:1196] T eb0f4fe6b8684f329725ac2f7d5428f2 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal/wals/eb0f4fe6b8684f329725ac2f7d5428f2
I20240503 02:14:22.478698 29719 catalog_manager.cc:4830] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): tablet eb0f4fe6b8684f329725ac2f7d5428f2 (table rebalancer_extra_table_34 [id=2274af2004634409884d3bcbca0009ca]) successfully deleted
I20240503 02:14:22.781183 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:22.798923 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:22.813992 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:22.865725 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:22.866870 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:22.867265 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:22.867542 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:22.868981 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:23.112942 567 consensus_queue.cc:237] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:14:23.115329 434 raft_consensus.cc:1261] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:23.115331 32640 raft_consensus.cc:1261] T ba97049157a646c780fa203d34cd4c82 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:23.115842 12351 consensus_queue.cc:1035] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:23.116096 12352 consensus_queue.cc:1035] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:23.119144 12352 raft_consensus.cc:2932] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:23.119669 434 raft_consensus.cc:2932] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:23.120007 32640 raft_consensus.cc:2932] T ba97049157a646c780fa203d34cd4c82 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:23.125808 29729 catalog_manager.cc:5484] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: config changed from index 2 to 3, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) evicted. New cstate: current_term: 1 leader_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:23.142668 30244 tablet_service.cc:1514] Processing DeleteTablet for tablet ba97049157a646c780fa203d34cd4c82 with delete_type TABLET_DATA_TOMBSTONED (TS 1f10f80f13c84e80b03eda7085a3b012 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:49856
I20240503 02:14:23.143307 12562 tablet_replica.cc:330] T ba97049157a646c780fa203d34cd4c82 P 1f10f80f13c84e80b03eda7085a3b012: stopping tablet replica
I20240503 02:14:23.143556 12562 raft_consensus.cc:2229] T ba97049157a646c780fa203d34cd4c82 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:23.143903 12562 raft_consensus.cc:2258] T ba97049157a646c780fa203d34cd4c82 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:23.144613 12562 ts_tablet_manager.cc:1875] T ba97049157a646c780fa203d34cd4c82 P 1f10f80f13c84e80b03eda7085a3b012: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:23.153226 12562 ts_tablet_manager.cc:1888] T ba97049157a646c780fa203d34cd4c82 P 1f10f80f13c84e80b03eda7085a3b012: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:14:23.153385 12562 log.cc:1196] T ba97049157a646c780fa203d34cd4c82 P 1f10f80f13c84e80b03eda7085a3b012: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/wals/ba97049157a646c780fa203d34cd4c82
I20240503 02:14:23.154139 29719 catalog_manager.cc:4830] TS 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): tablet ba97049157a646c780fa203d34cd4c82 (table rebalancer_extra_table_6 [id=53af41732fcb40aea2f1c6f0ac294bde]) successfully deleted
I20240503 02:14:23.443672 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:23.482887 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:23.491140 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:23.510434 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:23.599813 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:23.601264 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:23.604199 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:23.667388 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:23.787523 567 consensus_queue.cc:237] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:23.790702 32641 raft_consensus.cc:1261] T 9be5395489a3488da62f4d5b66facf3d P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:23.791163 30133 raft_consensus.cc:1261] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:23.791222 12352 consensus_queue.cc:1035] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:23.791664 12553 consensus_queue.cc:1035] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:23.793860 12352 raft_consensus.cc:2932] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:23.794402 32641 raft_consensus.cc:2932] T 9be5395489a3488da62f4d5b66facf3d P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:23.794507 30133 raft_consensus.cc:2932] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:23.800690 29731 catalog_manager.cc:5484] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 reported cstate change: config changed from index 2 to 3, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New cstate: current_term: 1 leader_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:23.812482 32755 tablet_service.cc:1514] Processing DeleteTablet for tablet 9be5395489a3488da62f4d5b66facf3d with delete_type TABLET_DATA_TOMBSTONED (TS 8a666e798fe8477d8dd1e51b385d5ead not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46360
I20240503 02:14:23.814318 12651 tablet_replica.cc:330] T 9be5395489a3488da62f4d5b66facf3d P 8a666e798fe8477d8dd1e51b385d5ead: stopping tablet replica
I20240503 02:14:23.814596 12651 raft_consensus.cc:2229] T 9be5395489a3488da62f4d5b66facf3d P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:23.814824 12651 raft_consensus.cc:2258] T 9be5395489a3488da62f4d5b66facf3d P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:23.815491 12651 ts_tablet_manager.cc:1875] T 9be5395489a3488da62f4d5b66facf3d P 8a666e798fe8477d8dd1e51b385d5ead: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:23.823483 12651 ts_tablet_manager.cc:1888] T 9be5395489a3488da62f4d5b66facf3d P 8a666e798fe8477d8dd1e51b385d5ead: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:14:23.823673 12651 log.cc:1196] T 9be5395489a3488da62f4d5b66facf3d P 8a666e798fe8477d8dd1e51b385d5ead: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal/wals/9be5395489a3488da62f4d5b66facf3d
I20240503 02:14:23.824714 29719 catalog_manager.cc:4830] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): tablet 9be5395489a3488da62f4d5b66facf3d (table rebalancer_extra_table_2 [id=27940b10e50e4a418d9e7155f5a25394]) successfully deleted
I20240503 02:14:23.944988 29873 consensus_queue.cc:237] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:23.948058 306 raft_consensus.cc:1261] T 4055b2f8d72041acaa81e6e98db9042c P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:23.948197 567 raft_consensus.cc:1261] T 4055b2f8d72041acaa81e6e98db9042c P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:14:23.948720 29808 consensus_peers.cc:485] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 -> Peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Couldn't send request to peer a9094240860e43bfb129193f3bfd5be6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4055b2f8d72041acaa81e6e98db9042c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:23.948786 12688 consensus_queue.cc:1035] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:23.949012 12411 consensus_queue.cc:1035] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:23.954018 12411 raft_consensus.cc:2932] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:23.954568 567 raft_consensus.cc:2932] T 4055b2f8d72041acaa81e6e98db9042c P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:23.960892 306 raft_consensus.cc:2932] T 4055b2f8d72041acaa81e6e98db9042c P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:23.961421 29730 catalog_manager.cc:5484] T 4055b2f8d72041acaa81e6e98db9042c P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:24.010375 307 consensus_queue.cc:237] T 331905034afd48c19eb2bc2e2ec40280 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
W20240503 02:14:24.014784 32711 consensus_peers.cc:485] T 331905034afd48c19eb2bc2e2ec40280 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Couldn't send request to peer 16440e365430405fb10366ab70e2b429. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 331905034afd48c19eb2bc2e2ec40280. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:24.014925 32640 raft_consensus.cc:1261] T 331905034afd48c19eb2bc2e2ec40280 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:24.015537 12710 consensus_queue.cc:1035] T 331905034afd48c19eb2bc2e2ec40280 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:24.015697 30004 raft_consensus.cc:1261] T 331905034afd48c19eb2bc2e2ec40280 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:24.016222 12532 consensus_queue.cc:1035] T 331905034afd48c19eb2bc2e2ec40280 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:24.026216 12532 raft_consensus.cc:2932] T 331905034afd48c19eb2bc2e2ec40280 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:24.026890 30004 raft_consensus.cc:2932] T 331905034afd48c19eb2bc2e2ec40280 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:24.031978 32640 raft_consensus.cc:2932] T 331905034afd48c19eb2bc2e2ec40280 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:24.033097 29731 catalog_manager.cc:5484] T 331905034afd48c19eb2bc2e2ec40280 P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: config changed from index -1 to 2, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) added. New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:24.073477 30004 consensus_queue.cc:237] T a52bee1fdc9d48b1a3463ecd62c8b44e P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:24.076165 434 raft_consensus.cc:1261] T a52bee1fdc9d48b1a3463ecd62c8b44e P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:14:24.076381 29939 consensus_peers.cc:485] T a52bee1fdc9d48b1a3463ecd62c8b44e P 155735c7f8b14997ac4c770aee2a9401 -> Peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Couldn't send request to peer a9094240860e43bfb129193f3bfd5be6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a52bee1fdc9d48b1a3463ecd62c8b44e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:24.076925 12528 consensus_queue.cc:1035] T a52bee1fdc9d48b1a3463ecd62c8b44e P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:24.077843 307 raft_consensus.cc:1261] T a52bee1fdc9d48b1a3463ecd62c8b44e P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:24.078450 12528 consensus_queue.cc:1035] T a52bee1fdc9d48b1a3463ecd62c8b44e P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:24.080909 12730 raft_consensus.cc:2932] T a52bee1fdc9d48b1a3463ecd62c8b44e P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:24.081498 307 raft_consensus.cc:2932] T a52bee1fdc9d48b1a3463ecd62c8b44e P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:24.085274 434 raft_consensus.cc:2932] T a52bee1fdc9d48b1a3463ecd62c8b44e P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:24.092216 29731 catalog_manager.cc:5484] T a52bee1fdc9d48b1a3463ecd62c8b44e P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:24.368968 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:24.374097 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:24.420383 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:24.436203 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:24.442781 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:24.483187 12790 ts_tablet_manager.cc:902] T 4055b2f8d72041acaa81e6e98db9042c P a9094240860e43bfb129193f3bfd5be6: Initiating tablet copy from peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:14:24.516583 12790 tablet_copy_client.cc:308] T 4055b2f8d72041acaa81e6e98db9042c P a9094240860e43bfb129193f3bfd5be6: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.1:36719
I20240503 02:14:24.517402 29895 tablet_copy_service.cc:140] P 16440e365430405fb10366ab70e2b429: Received BeginTabletCopySession request for tablet 4055b2f8d72041acaa81e6e98db9042c from peer a9094240860e43bfb129193f3bfd5be6 ({username='slave'} at 127.22.108.3:49185)
I20240503 02:14:24.517575 29895 tablet_copy_service.cc:161] P 16440e365430405fb10366ab70e2b429: Beginning new tablet copy session on tablet 4055b2f8d72041acaa81e6e98db9042c from peer a9094240860e43bfb129193f3bfd5be6 at {username='slave'} at 127.22.108.3:49185: session id = a9094240860e43bfb129193f3bfd5be6-4055b2f8d72041acaa81e6e98db9042c
I20240503 02:14:24.520226 29895 tablet_copy_source_session.cc:205] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:24.522096 12790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4055b2f8d72041acaa81e6e98db9042c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:24.526873 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:24.533768 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:24.535396 12790 tablet_copy_client.cc:783] T 4055b2f8d72041acaa81e6e98db9042c P a9094240860e43bfb129193f3bfd5be6: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:24.535643 12790 tablet_copy_client.cc:647] T 4055b2f8d72041acaa81e6e98db9042c P a9094240860e43bfb129193f3bfd5be6: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:24.549398 12790 tablet_copy_client.cc:523] T 4055b2f8d72041acaa81e6e98db9042c P a9094240860e43bfb129193f3bfd5be6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:24.553355 12790 tablet_bootstrap.cc:492] T 4055b2f8d72041acaa81e6e98db9042c P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:14:24.563194 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:24.615537 12790 tablet_bootstrap.cc:492] T 4055b2f8d72041acaa81e6e98db9042c P a9094240860e43bfb129193f3bfd5be6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:24.620443 12790 tablet_bootstrap.cc:492] T 4055b2f8d72041acaa81e6e98db9042c P a9094240860e43bfb129193f3bfd5be6: Bootstrap complete.
I20240503 02:14:24.620747 12790 ts_tablet_manager.cc:1367] T 4055b2f8d72041acaa81e6e98db9042c P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.067s user 0.022s sys 0.010s
I20240503 02:14:24.624321 12790 raft_consensus.cc:348] T 4055b2f8d72041acaa81e6e98db9042c P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:24.624562 12790 raft_consensus.cc:729] T 4055b2f8d72041acaa81e6e98db9042c P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:14:24.624795 12790 consensus_queue.cc:260] T 4055b2f8d72041acaa81e6e98db9042c P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:24.629577 12790 ts_tablet_manager.cc:1398] T 4055b2f8d72041acaa81e6e98db9042c P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 02:14:24.631476 29895 tablet_copy_service.cc:342] P 16440e365430405fb10366ab70e2b429: Request end of tablet copy session a9094240860e43bfb129193f3bfd5be6-4055b2f8d72041acaa81e6e98db9042c received from {username='slave'} at 127.22.108.3:49185
I20240503 02:14:24.631587 29895 tablet_copy_service.cc:433] P 16440e365430405fb10366ab70e2b429: ending tablet copy session a9094240860e43bfb129193f3bfd5be6-4055b2f8d72041acaa81e6e98db9042c on tablet 4055b2f8d72041acaa81e6e98db9042c with peer a9094240860e43bfb129193f3bfd5be6
I20240503 02:14:24.633666 12797 ts_tablet_manager.cc:902] T 331905034afd48c19eb2bc2e2ec40280 P 16440e365430405fb10366ab70e2b429: Initiating tablet copy from peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:14:24.686389 12797 tablet_copy_client.cc:308] T 331905034afd48c19eb2bc2e2ec40280 P 16440e365430405fb10366ab70e2b429: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.6:39975
I20240503 02:14:24.687021 327 tablet_copy_service.cc:140] P 8a666e798fe8477d8dd1e51b385d5ead: Received BeginTabletCopySession request for tablet 331905034afd48c19eb2bc2e2ec40280 from peer 16440e365430405fb10366ab70e2b429 ({username='slave'} at 127.22.108.1:40161)
I20240503 02:14:24.687179 327 tablet_copy_service.cc:161] P 8a666e798fe8477d8dd1e51b385d5ead: Beginning new tablet copy session on tablet 331905034afd48c19eb2bc2e2ec40280 from peer 16440e365430405fb10366ab70e2b429 at {username='slave'} at 127.22.108.1:40161: session id = 16440e365430405fb10366ab70e2b429-331905034afd48c19eb2bc2e2ec40280
I20240503 02:14:24.689294 327 tablet_copy_source_session.cc:205] T 331905034afd48c19eb2bc2e2ec40280 P 8a666e798fe8477d8dd1e51b385d5ead: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:24.690583 12797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 331905034afd48c19eb2bc2e2ec40280. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:24.696687 12790 ts_tablet_manager.cc:902] T a52bee1fdc9d48b1a3463ecd62c8b44e P a9094240860e43bfb129193f3bfd5be6: Initiating tablet copy from peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:14:24.696813 12797 tablet_copy_client.cc:783] T 331905034afd48c19eb2bc2e2ec40280 P 16440e365430405fb10366ab70e2b429: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:24.697011 12797 tablet_copy_client.cc:647] T 331905034afd48c19eb2bc2e2ec40280 P 16440e365430405fb10366ab70e2b429: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:24.699800 12797 tablet_copy_client.cc:523] T 331905034afd48c19eb2bc2e2ec40280 P 16440e365430405fb10366ab70e2b429: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:24.705943 12797 tablet_bootstrap.cc:492] T 331905034afd48c19eb2bc2e2ec40280 P 16440e365430405fb10366ab70e2b429: Bootstrap starting.
I20240503 02:14:24.730048 12790 tablet_copy_client.cc:308] T a52bee1fdc9d48b1a3463ecd62c8b44e P a9094240860e43bfb129193f3bfd5be6: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.2:37873
I20240503 02:14:24.731011 30026 tablet_copy_service.cc:140] P 155735c7f8b14997ac4c770aee2a9401: Received BeginTabletCopySession request for tablet a52bee1fdc9d48b1a3463ecd62c8b44e from peer a9094240860e43bfb129193f3bfd5be6 ({username='slave'} at 127.22.108.3:41369)
I20240503 02:14:24.731185 30026 tablet_copy_service.cc:161] P 155735c7f8b14997ac4c770aee2a9401: Beginning new tablet copy session on tablet a52bee1fdc9d48b1a3463ecd62c8b44e from peer a9094240860e43bfb129193f3bfd5be6 at {username='slave'} at 127.22.108.3:41369: session id = a9094240860e43bfb129193f3bfd5be6-a52bee1fdc9d48b1a3463ecd62c8b44e
I20240503 02:14:24.731928 12797 tablet_bootstrap.cc:492] T 331905034afd48c19eb2bc2e2ec40280 P 16440e365430405fb10366ab70e2b429: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:24.732230 12797 tablet_bootstrap.cc:492] T 331905034afd48c19eb2bc2e2ec40280 P 16440e365430405fb10366ab70e2b429: Bootstrap complete.
I20240503 02:14:24.732462 12797 ts_tablet_manager.cc:1367] T 331905034afd48c19eb2bc2e2ec40280 P 16440e365430405fb10366ab70e2b429: Time spent bootstrapping tablet: real 0.027s user 0.014s sys 0.010s
I20240503 02:14:24.733167 12797 raft_consensus.cc:348] T 331905034afd48c19eb2bc2e2ec40280 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:14:24.733424 12797 raft_consensus.cc:729] T 331905034afd48c19eb2bc2e2ec40280 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16440e365430405fb10366ab70e2b429, State: Initialized, Role: FOLLOWER
I20240503 02:14:24.733642 12797 consensus_queue.cc:260] T 331905034afd48c19eb2bc2e2ec40280 P 16440e365430405fb10366ab70e2b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:14:24.734163 30026 tablet_copy_source_session.cc:205] T a52bee1fdc9d48b1a3463ecd62c8b44e P 155735c7f8b14997ac4c770aee2a9401: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:24.734309 12797 ts_tablet_manager.cc:1398] T 331905034afd48c19eb2bc2e2ec40280 P 16440e365430405fb10366ab70e2b429: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:14:24.735594 327 tablet_copy_service.cc:342] P 8a666e798fe8477d8dd1e51b385d5ead: Request end of tablet copy session 16440e365430405fb10366ab70e2b429-331905034afd48c19eb2bc2e2ec40280 received from {username='slave'} at 127.22.108.1:40161
I20240503 02:14:24.735702 327 tablet_copy_service.cc:433] P 8a666e798fe8477d8dd1e51b385d5ead: ending tablet copy session 16440e365430405fb10366ab70e2b429-331905034afd48c19eb2bc2e2ec40280 on tablet 331905034afd48c19eb2bc2e2ec40280 with peer 16440e365430405fb10366ab70e2b429
I20240503 02:14:24.738850 12790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a52bee1fdc9d48b1a3463ecd62c8b44e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:24.744935 12790 tablet_copy_client.cc:783] T a52bee1fdc9d48b1a3463ecd62c8b44e P a9094240860e43bfb129193f3bfd5be6: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:24.745222 12790 tablet_copy_client.cc:647] T a52bee1fdc9d48b1a3463ecd62c8b44e P a9094240860e43bfb129193f3bfd5be6: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:24.747042 12790 tablet_copy_client.cc:523] T a52bee1fdc9d48b1a3463ecd62c8b44e P a9094240860e43bfb129193f3bfd5be6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:24.750062 12790 tablet_bootstrap.cc:492] T a52bee1fdc9d48b1a3463ecd62c8b44e P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:14:24.755147 12790 tablet_bootstrap.cc:492] T a52bee1fdc9d48b1a3463ecd62c8b44e P a9094240860e43bfb129193f3bfd5be6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:24.755548 12790 tablet_bootstrap.cc:492] T a52bee1fdc9d48b1a3463ecd62c8b44e P a9094240860e43bfb129193f3bfd5be6: Bootstrap complete.
I20240503 02:14:24.755791 12790 ts_tablet_manager.cc:1367] T a52bee1fdc9d48b1a3463ecd62c8b44e P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 02:14:24.756561 12790 raft_consensus.cc:348] T a52bee1fdc9d48b1a3463ecd62c8b44e P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:24.756772 12790 raft_consensus.cc:729] T a52bee1fdc9d48b1a3463ecd62c8b44e P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:14:24.757007 12790 consensus_queue.cc:260] T a52bee1fdc9d48b1a3463ecd62c8b44e P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:24.757799 12790 ts_tablet_manager.cc:1398] T a52bee1fdc9d48b1a3463ecd62c8b44e P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
I20240503 02:14:24.759457 30026 tablet_copy_service.cc:342] P 155735c7f8b14997ac4c770aee2a9401: Request end of tablet copy session a9094240860e43bfb129193f3bfd5be6-a52bee1fdc9d48b1a3463ecd62c8b44e received from {username='slave'} at 127.22.108.3:41369
I20240503 02:14:24.759573 30026 tablet_copy_service.cc:433] P 155735c7f8b14997ac4c770aee2a9401: ending tablet copy session a9094240860e43bfb129193f3bfd5be6-a52bee1fdc9d48b1a3463ecd62c8b44e on tablet a52bee1fdc9d48b1a3463ecd62c8b44e with peer a9094240860e43bfb129193f3bfd5be6
W20240503 02:14:24.952883 32709 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.4:33935, user_credentials={real_user=slave}} blocked reactor thread for 43520.3us
I20240503 02:14:25.035971 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:25.037528 29873 raft_consensus.cc:1203] T 331905034afd48c19eb2bc2e2ec40280 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:14:25.076426 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:25.083869 30133 raft_consensus.cc:1203] T 4055b2f8d72041acaa81e6e98db9042c P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:14:25.120282 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:25.122315 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:25.124874 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:25.187383 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:25.248502 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:25.309330 30133 raft_consensus.cc:1203] T a52bee1fdc9d48b1a3463ecd62c8b44e P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:14:25.313792 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:25.448892 29873 consensus_queue.cc:237] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:25.451100 567 raft_consensus.cc:1261] T 4055b2f8d72041acaa81e6e98db9042c P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:25.451157 30132 raft_consensus.cc:1261] T 4055b2f8d72041acaa81e6e98db9042c P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:25.451678 12688 consensus_queue.cc:1035] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:25.451885 12411 consensus_queue.cc:1035] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:25.455210 12688 raft_consensus.cc:2932] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:25.458117 30133 raft_consensus.cc:2932] T 4055b2f8d72041acaa81e6e98db9042c P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:25.464738 29731 catalog_manager.cc:5484] T 4055b2f8d72041acaa81e6e98db9042c P a9094240860e43bfb129193f3bfd5be6 reported cstate change: config changed from index 2 to 3, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:25.472838 567 raft_consensus.cc:2932] T 4055b2f8d72041acaa81e6e98db9042c P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:25.489946 32755 tablet_service.cc:1514] Processing DeleteTablet for tablet 4055b2f8d72041acaa81e6e98db9042c with delete_type TABLET_DATA_TOMBSTONED (TS 8a666e798fe8477d8dd1e51b385d5ead not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46360
I20240503 02:14:25.494755 12913 tablet_replica.cc:330] T 4055b2f8d72041acaa81e6e98db9042c P 8a666e798fe8477d8dd1e51b385d5ead: stopping tablet replica
I20240503 02:14:25.494972 12913 raft_consensus.cc:2229] T 4055b2f8d72041acaa81e6e98db9042c P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:25.498696 12913 raft_consensus.cc:2258] T 4055b2f8d72041acaa81e6e98db9042c P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:25.499486 12913 ts_tablet_manager.cc:1875] T 4055b2f8d72041acaa81e6e98db9042c P 8a666e798fe8477d8dd1e51b385d5ead: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:25.504973 12913 ts_tablet_manager.cc:1888] T 4055b2f8d72041acaa81e6e98db9042c P 8a666e798fe8477d8dd1e51b385d5ead: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:14:25.505095 12913 log.cc:1196] T 4055b2f8d72041acaa81e6e98db9042c P 8a666e798fe8477d8dd1e51b385d5ead: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal/wals/4055b2f8d72041acaa81e6e98db9042c
I20240503 02:14:25.505781 29719 catalog_manager.cc:4830] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): tablet 4055b2f8d72041acaa81e6e98db9042c (table rebalancer_extra_table_32 [id=567dfe3064ef40a99eede42015bd1a35]) successfully deleted
I20240503 02:14:25.756031 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:25.756857 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:25.792739 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:25.798069 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:25.801146 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:25.809736 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:25.810557 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:25.837776 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:26.023538 434 consensus_queue.cc:237] T 88f696d03330448d879ed7b60364c28c P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:14:26.025344 29873 raft_consensus.cc:1261] T 88f696d03330448d879ed7b60364c28c P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:26.025475 30264 raft_consensus.cc:1261] T 88f696d03330448d879ed7b60364c28c P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:26.025935 12789 consensus_queue.cc:1035] T 88f696d03330448d879ed7b60364c28c P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:26.026227 12861 consensus_queue.cc:1035] T 88f696d03330448d879ed7b60364c28c P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:26.029407 12789 raft_consensus.cc:2932] T 88f696d03330448d879ed7b60364c28c P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:26.032698 30264 raft_consensus.cc:2932] T 88f696d03330448d879ed7b60364c28c P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:26.033623 29873 raft_consensus.cc:2932] T 88f696d03330448d879ed7b60364c28c P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:26.036260 29730 catalog_manager.cc:5484] T 88f696d03330448d879ed7b60364c28c P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:26.058154 32619 tablet_service.cc:1514] Processing DeleteTablet for tablet 88f696d03330448d879ed7b60364c28c with delete_type TABLET_DATA_TOMBSTONED (TS 9a4d92bc25374fb3960fde7d10be309d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:38200
I20240503 02:14:26.059170 12990 tablet_replica.cc:330] T 88f696d03330448d879ed7b60364c28c P 9a4d92bc25374fb3960fde7d10be309d: stopping tablet replica
I20240503 02:14:26.059448 12990 raft_consensus.cc:2229] T 88f696d03330448d879ed7b60364c28c P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:26.061559 12990 raft_consensus.cc:2258] T 88f696d03330448d879ed7b60364c28c P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:26.062556 12990 ts_tablet_manager.cc:1875] T 88f696d03330448d879ed7b60364c28c P 9a4d92bc25374fb3960fde7d10be309d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:26.071754 12990 ts_tablet_manager.cc:1888] T 88f696d03330448d879ed7b60364c28c P 9a4d92bc25374fb3960fde7d10be309d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:14:26.071879 12990 log.cc:1196] T 88f696d03330448d879ed7b60364c28c P 9a4d92bc25374fb3960fde7d10be309d: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal/wals/88f696d03330448d879ed7b60364c28c
I20240503 02:14:26.072594 29719 catalog_manager.cc:4830] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): tablet 88f696d03330448d879ed7b60364c28c (table rebalancer_extra_table_24 [id=782c508495e74448b6859bfc4ff35de0]) successfully deleted
I20240503 02:14:26.360634 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:26.378515 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:26.389047 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:26.389380 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:26.429697 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:26.450153 30052 maintenance_manager.cc:419] P 155735c7f8b14997ac4c770aee2a9401: Scheduling FlushMRSOp(46d608d84863424e8c65feac7c2e835f): perf score=0.033334
W20240503 02:14:26.504182 504 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.1:36719, user_credentials={real_user=slave}} blocked reactor thread for 94807.3us
I20240503 02:14:26.514492 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:26.558336 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:26.568013 29921 maintenance_manager.cc:419] P 16440e365430405fb10366ab70e2b429: Scheduling FlushMRSOp(e80b43971dc246d9bfd5e1c603f8415c): perf score=0.033362
I20240503 02:14:26.579233 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:26.589823 30182 maintenance_manager.cc:419] P a9094240860e43bfb129193f3bfd5be6: Scheduling FlushMRSOp(e80b43971dc246d9bfd5e1c603f8415c): perf score=0.033367
I20240503 02:14:26.672884 30312 maintenance_manager.cc:419] P 1f10f80f13c84e80b03eda7085a3b012: Scheduling FlushMRSOp(96de1c26ae884ea18b63b328db040ba1): perf score=0.033391
I20240503 02:14:26.839979 30203 maintenance_manager.cc:643] P 1f10f80f13c84e80b03eda7085a3b012: FlushMRSOp(96de1c26ae884ea18b63b328db040ba1) complete. Timing: real 0.166s user 0.046s sys 0.035s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":20235,"dirs.queue_time_us":12189,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":1833,"drs_written":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4,"lbm_write_time_us":832,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":50528,"threads_started":3,"wal-append.queue_time_us":6334}
I20240503 02:14:26.843271 30312 maintenance_manager.cc:419] P 1f10f80f13c84e80b03eda7085a3b012: Scheduling UndoDeltaBlockGCOp(96de1c26ae884ea18b63b328db040ba1): 4581 bytes on disk
I20240503 02:14:26.844170 30203 maintenance_manager.cc:643] P 1f10f80f13c84e80b03eda7085a3b012: UndoDeltaBlockGCOp(96de1c26ae884ea18b63b328db040ba1) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
I20240503 02:14:26.899966 29943 maintenance_manager.cc:643] P 155735c7f8b14997ac4c770aee2a9401: FlushMRSOp(46d608d84863424e8c65feac7c2e835f) complete. Timing: real 0.449s user 0.064s sys 0.064s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":15697,"dirs.queue_time_us":20972,"dirs.run_cpu_time_us":231,"dirs.run_wall_time_us":994,"drs_written":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1020,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":84725,"threads_started":3,"wal-append.queue_time_us":42090}
I20240503 02:14:26.903797 30052 maintenance_manager.cc:419] P 155735c7f8b14997ac4c770aee2a9401: Scheduling UndoDeltaBlockGCOp(46d608d84863424e8c65feac7c2e835f): 4838 bytes on disk
I20240503 02:14:26.904774 29943 maintenance_manager.cc:643] P 155735c7f8b14997ac4c770aee2a9401: UndoDeltaBlockGCOp(46d608d84863424e8c65feac7c2e835f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240503 02:14:27.091806 30006 consensus_queue.cc:237] T a52bee1fdc9d48b1a3463ecd62c8b44e P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:27.094036 435 raft_consensus.cc:1261] T a52bee1fdc9d48b1a3463ecd62c8b44e P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:27.094312 30132 raft_consensus.cc:1261] T a52bee1fdc9d48b1a3463ecd62c8b44e P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:27.094542 13038 consensus_queue.cc:1035] T a52bee1fdc9d48b1a3463ecd62c8b44e P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:27.098459 13038 consensus_queue.cc:1035] T a52bee1fdc9d48b1a3463ecd62c8b44e P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:27.101069 13039 raft_consensus.cc:2932] T a52bee1fdc9d48b1a3463ecd62c8b44e P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:27.107311 435 raft_consensus.cc:2932] T a52bee1fdc9d48b1a3463ecd62c8b44e P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:27.116942 29729 catalog_manager.cc:5484] T a52bee1fdc9d48b1a3463ecd62c8b44e P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: config changed from index 2 to 3, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:27.122510 30133 raft_consensus.cc:2932] T a52bee1fdc9d48b1a3463ecd62c8b44e P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:27.197844 30073 maintenance_manager.cc:643] P a9094240860e43bfb129193f3bfd5be6: FlushMRSOp(e80b43971dc246d9bfd5e1c603f8415c) complete. Timing: real 0.607s user 0.087s sys 0.023s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":13467,"dirs.queue_time_us":11739,"dirs.run_cpu_time_us":265,"dirs.run_wall_time_us":15137,"drs_written":1,"lbm_read_time_us":155,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2311,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":95153,"threads_started":3,"wal-append.queue_time_us":49906}
I20240503 02:14:27.199740 32755 tablet_service.cc:1514] Processing DeleteTablet for tablet a52bee1fdc9d48b1a3463ecd62c8b44e with delete_type TABLET_DATA_TOMBSTONED (TS 8a666e798fe8477d8dd1e51b385d5ead not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46360
I20240503 02:14:27.201977 30182 maintenance_manager.cc:419] P a9094240860e43bfb129193f3bfd5be6: Scheduling UndoDeltaBlockGCOp(e80b43971dc246d9bfd5e1c603f8415c): 4924 bytes on disk
I20240503 02:14:27.202661 13104 tablet_replica.cc:330] T a52bee1fdc9d48b1a3463ecd62c8b44e P 8a666e798fe8477d8dd1e51b385d5ead: stopping tablet replica
I20240503 02:14:27.202925 13104 raft_consensus.cc:2229] T a52bee1fdc9d48b1a3463ecd62c8b44e P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:27.203194 13104 raft_consensus.cc:2258] T a52bee1fdc9d48b1a3463ecd62c8b44e P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:27.203902 13104 ts_tablet_manager.cc:1875] T a52bee1fdc9d48b1a3463ecd62c8b44e P 8a666e798fe8477d8dd1e51b385d5ead: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:27.194944 29812 maintenance_manager.cc:643] P 16440e365430405fb10366ab70e2b429: FlushMRSOp(e80b43971dc246d9bfd5e1c603f8415c) complete. Timing: real 0.626s user 0.088s sys 0.017s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":15709,"dirs.queue_time_us":21788,"dirs.run_cpu_time_us":245,"dirs.run_wall_time_us":7176,"drs_written":1,"lbm_read_time_us":128,"lbm_reads_lt_1ms":4,"lbm_write_time_us":8430,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":689,"thread_start_us":58107,"threads_started":3,"wal-append.queue_time_us":6296}
I20240503 02:14:27.209492 30073 maintenance_manager.cc:643] P a9094240860e43bfb129193f3bfd5be6: UndoDeltaBlockGCOp(e80b43971dc246d9bfd5e1c603f8415c) complete. Timing: real 0.003s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":2852,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3}
I20240503 02:14:27.211192 29921 maintenance_manager.cc:419] P 16440e365430405fb10366ab70e2b429: Scheduling UndoDeltaBlockGCOp(e80b43971dc246d9bfd5e1c603f8415c): 4924 bytes on disk
I20240503 02:14:27.213564 30182 maintenance_manager.cc:419] P a9094240860e43bfb129193f3bfd5be6: Scheduling FlushMRSOp(96de1c26ae884ea18b63b328db040ba1): perf score=0.033537
I20240503 02:14:27.211349 13104 ts_tablet_manager.cc:1888] T a52bee1fdc9d48b1a3463ecd62c8b44e P 8a666e798fe8477d8dd1e51b385d5ead: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:14:27.214068 13104 log.cc:1196] T a52bee1fdc9d48b1a3463ecd62c8b44e P 8a666e798fe8477d8dd1e51b385d5ead: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal/wals/a52bee1fdc9d48b1a3463ecd62c8b44e
I20240503 02:14:27.215065 29719 catalog_manager.cc:4830] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): tablet a52bee1fdc9d48b1a3463ecd62c8b44e (table rebalancer_extra_table_10 [id=81455612cf1243e09bae574807b9303b]) successfully deleted
I20240503 02:14:27.220168 29812 maintenance_manager.cc:643] P 16440e365430405fb10366ab70e2b429: UndoDeltaBlockGCOp(e80b43971dc246d9bfd5e1c603f8415c) complete. Timing: real 0.003s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":2745,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3}
I20240503 02:14:27.224004 29921 maintenance_manager.cc:419] P 16440e365430405fb10366ab70e2b429: Scheduling FlushMRSOp(46d608d84863424e8c65feac7c2e835f): perf score=0.033539
I20240503 02:14:27.479802 30073 maintenance_manager.cc:643] P a9094240860e43bfb129193f3bfd5be6: FlushMRSOp(96de1c26ae884ea18b63b328db040ba1) complete. Timing: real 0.264s user 0.038s sys 0.016s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":1660,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1042,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":29445,"threads_started":1,"wal-append.queue_time_us":17416}
I20240503 02:14:27.487912 30182 maintenance_manager.cc:419] P a9094240860e43bfb129193f3bfd5be6: Scheduling UndoDeltaBlockGCOp(96de1c26ae884ea18b63b328db040ba1): 4581 bytes on disk
I20240503 02:14:27.489007 30073 maintenance_manager.cc:643] P a9094240860e43bfb129193f3bfd5be6: UndoDeltaBlockGCOp(96de1c26ae884ea18b63b328db040ba1) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
I20240503 02:14:27.513406 29812 maintenance_manager.cc:643] P 16440e365430405fb10366ab70e2b429: FlushMRSOp(46d608d84863424e8c65feac7c2e835f) complete. Timing: real 0.287s user 0.045s sys 0.013s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":2462,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":1482,"drs_written":1,"lbm_read_time_us":5697,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3,"lbm_write_time_us":2758,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":12336,"threads_started":1,"wal-append.queue_time_us":12253}
I20240503 02:14:27.530218 29921 maintenance_manager.cc:419] P 16440e365430405fb10366ab70e2b429: Scheduling UndoDeltaBlockGCOp(46d608d84863424e8c65feac7c2e835f): 4838 bytes on disk
I20240503 02:14:27.543411 29812 maintenance_manager.cc:643] P 16440e365430405fb10366ab70e2b429: UndoDeltaBlockGCOp(46d608d84863424e8c65feac7c2e835f) complete. Timing: real 0.003s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":131,"lbm_reads_lt_1ms":4}
I20240503 02:14:27.823076 435 consensus_queue.cc:237] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
W20240503 02:14:27.833709 370 consensus_peers.cc:485] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 -> Peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Couldn't send request to peer a9094240860e43bfb129193f3bfd5be6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2baf70e0e1894cea9fdeb92279bc0cff. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:27.834779 306 raft_consensus.cc:1261] T 2baf70e0e1894cea9fdeb92279bc0cff P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:27.836043 12861 consensus_queue.cc:1035] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:27.843093 32641 raft_consensus.cc:1261] T 2baf70e0e1894cea9fdeb92279bc0cff P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:27.843725 12861 consensus_queue.cc:1035] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:27.859114 12861 raft_consensus.cc:2932] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:27.862387 32640 raft_consensus.cc:2932] T 2baf70e0e1894cea9fdeb92279bc0cff P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:27.870667 306 raft_consensus.cc:2932] T 2baf70e0e1894cea9fdeb92279bc0cff P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:27.872431 29732 catalog_manager.cc:5484] T 2baf70e0e1894cea9fdeb92279bc0cff P 9a4d92bc25374fb3960fde7d10be309d reported cstate change: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:28.140226 30006 consensus_queue.cc:237] T a6505f95a1c547748fb5cb7a4918b129 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
W20240503 02:14:28.147707 29939 consensus_peers.cc:485] T a6505f95a1c547748fb5cb7a4918b129 P 155735c7f8b14997ac4c770aee2a9401 -> Peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Couldn't send request to peer a9094240860e43bfb129193f3bfd5be6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a6505f95a1c547748fb5cb7a4918b129. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:28.149080 435 raft_consensus.cc:1261] T a6505f95a1c547748fb5cb7a4918b129 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:28.150063 13039 consensus_queue.cc:1035] T a6505f95a1c547748fb5cb7a4918b129 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:28.150804 306 raft_consensus.cc:1261] T a6505f95a1c547748fb5cb7a4918b129 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:28.152333 13039 consensus_queue.cc:1035] T a6505f95a1c547748fb5cb7a4918b129 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:28.175000 306 raft_consensus.cc:2932] T a6505f95a1c547748fb5cb7a4918b129 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:28.164664 13039 raft_consensus.cc:2932] T a6505f95a1c547748fb5cb7a4918b129 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:28.201632 29729 catalog_manager.cc:5484] T a6505f95a1c547748fb5cb7a4918b129 P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:28.262158 435 raft_consensus.cc:2932] T a6505f95a1c547748fb5cb7a4918b129 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:28.432031 434 consensus_queue.cc:237] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:28.437342 13192 ts_tablet_manager.cc:902] T 2baf70e0e1894cea9fdeb92279bc0cff P a9094240860e43bfb129193f3bfd5be6: Initiating tablet copy from peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:14:28.438573 32641 raft_consensus.cc:1261] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:28.438730 567 raft_consensus.cc:1261] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:28.439352 13177 consensus_queue.cc:1035] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:28.439785 12789 consensus_queue.cc:1035] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:14:28.440662 370 consensus_peers.cc:485] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 -> Peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Couldn't send request to peer a9094240860e43bfb129193f3bfd5be6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 737a39a4f65b4e9ab2ce65c0d7f62dcb. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:28.455760 13177 raft_consensus.cc:2932] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:28.461024 32641 raft_consensus.cc:2932] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:28.471158 29728 catalog_manager.cc:5484] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P 9a4d92bc25374fb3960fde7d10be309d reported cstate change: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:28.480974 13192 tablet_copy_client.cc:308] T 2baf70e0e1894cea9fdeb92279bc0cff P a9094240860e43bfb129193f3bfd5be6: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.7:44241
I20240503 02:14:28.481638 457 tablet_copy_service.cc:140] P fbbc2be6382e47908ecb7c68c367bd26: Received BeginTabletCopySession request for tablet 2baf70e0e1894cea9fdeb92279bc0cff from peer a9094240860e43bfb129193f3bfd5be6 ({username='slave'} at 127.22.108.3:42051)
I20240503 02:14:28.481820 457 tablet_copy_service.cc:161] P fbbc2be6382e47908ecb7c68c367bd26: Beginning new tablet copy session on tablet 2baf70e0e1894cea9fdeb92279bc0cff from peer a9094240860e43bfb129193f3bfd5be6 at {username='slave'} at 127.22.108.3:42051: session id = a9094240860e43bfb129193f3bfd5be6-2baf70e0e1894cea9fdeb92279bc0cff
I20240503 02:14:28.481877 565 raft_consensus.cc:2932] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:28.483668 457 tablet_copy_source_session.cc:205] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:28.485184 13192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2baf70e0e1894cea9fdeb92279bc0cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:28.491391 13192 tablet_copy_client.cc:783] T 2baf70e0e1894cea9fdeb92279bc0cff P a9094240860e43bfb129193f3bfd5be6: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:28.491683 13192 tablet_copy_client.cc:647] T 2baf70e0e1894cea9fdeb92279bc0cff P a9094240860e43bfb129193f3bfd5be6: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:28.494534 13192 tablet_copy_client.cc:523] T 2baf70e0e1894cea9fdeb92279bc0cff P a9094240860e43bfb129193f3bfd5be6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:28.497551 13192 tablet_bootstrap.cc:492] T 2baf70e0e1894cea9fdeb92279bc0cff P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:14:28.513084 13192 tablet_bootstrap.cc:492] T 2baf70e0e1894cea9fdeb92279bc0cff P a9094240860e43bfb129193f3bfd5be6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:28.513550 13192 tablet_bootstrap.cc:492] T 2baf70e0e1894cea9fdeb92279bc0cff P a9094240860e43bfb129193f3bfd5be6: Bootstrap complete.
I20240503 02:14:28.513788 13192 ts_tablet_manager.cc:1367] T 2baf70e0e1894cea9fdeb92279bc0cff P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.010s
I20240503 02:14:28.514390 13192 raft_consensus.cc:348] T 2baf70e0e1894cea9fdeb92279bc0cff P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:28.514627 13192 raft_consensus.cc:729] T 2baf70e0e1894cea9fdeb92279bc0cff P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:14:28.514878 13192 consensus_queue.cc:260] T 2baf70e0e1894cea9fdeb92279bc0cff P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:28.515429 13192 ts_tablet_manager.cc:1398] T 2baf70e0e1894cea9fdeb92279bc0cff P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:14:28.516417 457 tablet_copy_service.cc:342] P fbbc2be6382e47908ecb7c68c367bd26: Request end of tablet copy session a9094240860e43bfb129193f3bfd5be6-2baf70e0e1894cea9fdeb92279bc0cff received from {username='slave'} at 127.22.108.3:42051
I20240503 02:14:28.516548 457 tablet_copy_service.cc:433] P fbbc2be6382e47908ecb7c68c367bd26: ending tablet copy session a9094240860e43bfb129193f3bfd5be6-2baf70e0e1894cea9fdeb92279bc0cff on tablet 2baf70e0e1894cea9fdeb92279bc0cff with peer a9094240860e43bfb129193f3bfd5be6
I20240503 02:14:28.589592 13192 ts_tablet_manager.cc:902] T a6505f95a1c547748fb5cb7a4918b129 P a9094240860e43bfb129193f3bfd5be6: Initiating tablet copy from peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:14:28.591568 13192 tablet_copy_client.cc:308] T a6505f95a1c547748fb5cb7a4918b129 P a9094240860e43bfb129193f3bfd5be6: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.2:37873
I20240503 02:14:28.592586 30026 tablet_copy_service.cc:140] P 155735c7f8b14997ac4c770aee2a9401: Received BeginTabletCopySession request for tablet a6505f95a1c547748fb5cb7a4918b129 from peer a9094240860e43bfb129193f3bfd5be6 ({username='slave'} at 127.22.108.3:41369)
I20240503 02:14:28.592758 30026 tablet_copy_service.cc:161] P 155735c7f8b14997ac4c770aee2a9401: Beginning new tablet copy session on tablet a6505f95a1c547748fb5cb7a4918b129 from peer a9094240860e43bfb129193f3bfd5be6 at {username='slave'} at 127.22.108.3:41369: session id = a9094240860e43bfb129193f3bfd5be6-a6505f95a1c547748fb5cb7a4918b129
I20240503 02:14:28.595225 30026 tablet_copy_source_session.cc:205] T a6505f95a1c547748fb5cb7a4918b129 P 155735c7f8b14997ac4c770aee2a9401: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:28.596350 13192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6505f95a1c547748fb5cb7a4918b129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:28.602201 13192 tablet_copy_client.cc:783] T a6505f95a1c547748fb5cb7a4918b129 P a9094240860e43bfb129193f3bfd5be6: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:28.602468 13192 tablet_copy_client.cc:647] T a6505f95a1c547748fb5cb7a4918b129 P a9094240860e43bfb129193f3bfd5be6: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:28.603927 13192 tablet_copy_client.cc:523] T a6505f95a1c547748fb5cb7a4918b129 P a9094240860e43bfb129193f3bfd5be6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:28.610443 13192 tablet_bootstrap.cc:492] T a6505f95a1c547748fb5cb7a4918b129 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:14:28.622303 13192 tablet_bootstrap.cc:492] T a6505f95a1c547748fb5cb7a4918b129 P a9094240860e43bfb129193f3bfd5be6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:28.622704 13192 tablet_bootstrap.cc:492] T a6505f95a1c547748fb5cb7a4918b129 P a9094240860e43bfb129193f3bfd5be6: Bootstrap complete.
I20240503 02:14:28.622951 13192 ts_tablet_manager.cc:1367] T a6505f95a1c547748fb5cb7a4918b129 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20240503 02:14:28.623600 13192 raft_consensus.cc:348] T a6505f95a1c547748fb5cb7a4918b129 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:28.623867 13192 raft_consensus.cc:729] T a6505f95a1c547748fb5cb7a4918b129 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:14:28.624176 13192 consensus_queue.cc:260] T a6505f95a1c547748fb5cb7a4918b129 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:28.624769 13192 ts_tablet_manager.cc:1398] T a6505f95a1c547748fb5cb7a4918b129 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:14:28.626973 30026 tablet_copy_service.cc:342] P 155735c7f8b14997ac4c770aee2a9401: Request end of tablet copy session a9094240860e43bfb129193f3bfd5be6-a6505f95a1c547748fb5cb7a4918b129 received from {username='slave'} at 127.22.108.3:41369
I20240503 02:14:28.627090 30026 tablet_copy_service.cc:433] P 155735c7f8b14997ac4c770aee2a9401: ending tablet copy session a9094240860e43bfb129193f3bfd5be6-a6505f95a1c547748fb5cb7a4918b129 on tablet a6505f95a1c547748fb5cb7a4918b129 with peer a9094240860e43bfb129193f3bfd5be6
I20240503 02:14:28.760533 30182 maintenance_manager.cc:419] P a9094240860e43bfb129193f3bfd5be6: Scheduling FlushMRSOp(7527cc5931e24fa180efedbf81438efd): perf score=0.033340
I20240503 02:14:28.787420 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:28.812083 29921 maintenance_manager.cc:419] P 16440e365430405fb10366ab70e2b429: Scheduling FlushMRSOp(35a5a7f4fe184b9fadc94c98b61deaf0): perf score=0.033357
I20240503 02:14:28.820544 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:28.843313 30073 maintenance_manager.cc:643] P a9094240860e43bfb129193f3bfd5be6: FlushMRSOp(7527cc5931e24fa180efedbf81438efd) complete. Timing: real 0.082s user 0.037s sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":362,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":4108,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":869,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":6985,"threads_started":2,"wal-append.queue_time_us":6152}
I20240503 02:14:28.846017 30182 maintenance_manager.cc:419] P a9094240860e43bfb129193f3bfd5be6: Scheduling UndoDeltaBlockGCOp(7527cc5931e24fa180efedbf81438efd): 4584 bytes on disk
I20240503 02:14:28.846642 30073 maintenance_manager.cc:643] P a9094240860e43bfb129193f3bfd5be6: UndoDeltaBlockGCOp(7527cc5931e24fa180efedbf81438efd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20240503 02:14:28.859704 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:28.867130 30312 maintenance_manager.cc:419] P 1f10f80f13c84e80b03eda7085a3b012: Scheduling FlushMRSOp(683d4041910f4355910e188c0b7df6c7): perf score=0.033371
I20240503 02:14:28.867458 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:28.869248 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:28.872481 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:28.874619 13192 ts_tablet_manager.cc:902] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P a9094240860e43bfb129193f3bfd5be6: Initiating tablet copy from peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:14:28.882961 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:28.920758 13192 tablet_copy_client.cc:308] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P a9094240860e43bfb129193f3bfd5be6: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.7:44241
I20240503 02:14:28.920948 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:28.924415 457 tablet_copy_service.cc:140] P fbbc2be6382e47908ecb7c68c367bd26: Received BeginTabletCopySession request for tablet 737a39a4f65b4e9ab2ce65c0d7f62dcb from peer a9094240860e43bfb129193f3bfd5be6 ({username='slave'} at 127.22.108.3:42051)
I20240503 02:14:28.924594 457 tablet_copy_service.cc:161] P fbbc2be6382e47908ecb7c68c367bd26: Beginning new tablet copy session on tablet 737a39a4f65b4e9ab2ce65c0d7f62dcb from peer a9094240860e43bfb129193f3bfd5be6 at {username='slave'} at 127.22.108.3:42051: session id = a9094240860e43bfb129193f3bfd5be6-737a39a4f65b4e9ab2ce65c0d7f62dcb
I20240503 02:14:28.927382 457 tablet_copy_source_session.cc:205] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:28.929488 13192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 737a39a4f65b4e9ab2ce65c0d7f62dcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:28.932724 30052 maintenance_manager.cc:419] P 155735c7f8b14997ac4c770aee2a9401: Scheduling FlushMRSOp(35a5a7f4fe184b9fadc94c98b61deaf0): perf score=0.033394
I20240503 02:14:28.943845 13192 tablet_copy_client.cc:783] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P a9094240860e43bfb129193f3bfd5be6: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:28.944543 13192 tablet_copy_client.cc:647] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P a9094240860e43bfb129193f3bfd5be6: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:28.947145 13192 tablet_copy_client.cc:523] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P a9094240860e43bfb129193f3bfd5be6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:28.951809 29812 maintenance_manager.cc:643] P 16440e365430405fb10366ab70e2b429: FlushMRSOp(35a5a7f4fe184b9fadc94c98b61deaf0) complete. Timing: real 0.139s user 0.054s sys 0.034s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":33919,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":15056,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":953,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":41755,"threads_started":2,"wal-append.queue_time_us":7804}
I20240503 02:14:28.954294 29921 maintenance_manager.cc:419] P 16440e365430405fb10366ab70e2b429: Scheduling FlushMRSOp(7527cc5931e24fa180efedbf81438efd): perf score=0.033390
I20240503 02:14:28.962134 13192 tablet_bootstrap.cc:492] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:14:28.977742 30132 raft_consensus.cc:1203] T a6505f95a1c547748fb5cb7a4918b129 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:14:28.983343 30132 raft_consensus.cc:1203] T 2baf70e0e1894cea9fdeb92279bc0cff P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:14:28.989344 13192 tablet_bootstrap.cc:492] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P a9094240860e43bfb129193f3bfd5be6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:28.989871 13192 tablet_bootstrap.cc:492] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P a9094240860e43bfb129193f3bfd5be6: Bootstrap complete.
I20240503 02:14:28.990687 13192 ts_tablet_manager.cc:1367] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.001s
I20240503 02:14:28.991892 13192 raft_consensus.cc:348] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:28.992300 13192 raft_consensus.cc:729] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:14:28.992573 13192 consensus_queue.cc:260] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:28.997138 13192 ts_tablet_manager.cc:1398] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:14:28.997793 457 tablet_copy_service.cc:342] P fbbc2be6382e47908ecb7c68c367bd26: Request end of tablet copy session a9094240860e43bfb129193f3bfd5be6-737a39a4f65b4e9ab2ce65c0d7f62dcb received from {username='slave'} at 127.22.108.3:42051
I20240503 02:14:28.997921 457 tablet_copy_service.cc:433] P fbbc2be6382e47908ecb7c68c367bd26: ending tablet copy session a9094240860e43bfb129193f3bfd5be6-737a39a4f65b4e9ab2ce65c0d7f62dcb on tablet 737a39a4f65b4e9ab2ce65c0d7f62dcb with peer a9094240860e43bfb129193f3bfd5be6
I20240503 02:14:29.004726 29812 maintenance_manager.cc:643] P 16440e365430405fb10366ab70e2b429: FlushMRSOp(7527cc5931e24fa180efedbf81438efd) complete. Timing: real 0.050s user 0.044s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":1794,"dirs.run_cpu_time_us":190,"dirs.run_wall_time_us":1474,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":948,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":412,"threads_started":1,"wal-append.queue_time_us":505}
I20240503 02:14:29.006764 29921 maintenance_manager.cc:419] P 16440e365430405fb10366ab70e2b429: Scheduling UndoDeltaBlockGCOp(35a5a7f4fe184b9fadc94c98b61deaf0): 4923 bytes on disk
I20240503 02:14:29.007318 29812 maintenance_manager.cc:643] P 16440e365430405fb10366ab70e2b429: UndoDeltaBlockGCOp(35a5a7f4fe184b9fadc94c98b61deaf0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240503 02:14:29.009271 29921 maintenance_manager.cc:419] P 16440e365430405fb10366ab70e2b429: Scheduling UndoDeltaBlockGCOp(7527cc5931e24fa180efedbf81438efd): 4584 bytes on disk
I20240503 02:14:29.009784 29812 maintenance_manager.cc:643] P 16440e365430405fb10366ab70e2b429: UndoDeltaBlockGCOp(7527cc5931e24fa180efedbf81438efd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4}
I20240503 02:14:29.143066 29943 maintenance_manager.cc:643] P 155735c7f8b14997ac4c770aee2a9401: FlushMRSOp(35a5a7f4fe184b9fadc94c98b61deaf0) complete. Timing: real 0.210s user 0.069s sys 0.041s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":11723,"dirs.run_cpu_time_us":222,"dirs.run_wall_time_us":3163,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":794,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":69988,"threads_started":2,"wal-append.queue_time_us":58446}
I20240503 02:14:29.145846 30052 maintenance_manager.cc:419] P 155735c7f8b14997ac4c770aee2a9401: Scheduling UndoDeltaBlockGCOp(35a5a7f4fe184b9fadc94c98b61deaf0): 4923 bytes on disk
I20240503 02:14:29.146983 29943 maintenance_manager.cc:643] P 155735c7f8b14997ac4c770aee2a9401: UndoDeltaBlockGCOp(35a5a7f4fe184b9fadc94c98b61deaf0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20240503 02:14:29.151289 30052 maintenance_manager.cc:419] P 155735c7f8b14997ac4c770aee2a9401: Scheduling FlushMRSOp(683d4041910f4355910e188c0b7df6c7): perf score=0.033450
I20240503 02:14:29.285987 29943 maintenance_manager.cc:643] P 155735c7f8b14997ac4c770aee2a9401: FlushMRSOp(683d4041910f4355910e188c0b7df6c7) complete. Timing: real 0.134s user 0.047s sys 0.022s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":160,"dirs.run_wall_time_us":1311,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":867,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":6649,"threads_started":1,"wal-append.queue_time_us":6562}
I20240503 02:14:29.288820 30052 maintenance_manager.cc:419] P 155735c7f8b14997ac4c770aee2a9401: Scheduling UndoDeltaBlockGCOp(683d4041910f4355910e188c0b7df6c7): 4839 bytes on disk
I20240503 02:14:29.289917 29943 maintenance_manager.cc:643] P 155735c7f8b14997ac4c770aee2a9401: UndoDeltaBlockGCOp(683d4041910f4355910e188c0b7df6c7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
I20240503 02:14:29.296707 30203 maintenance_manager.cc:643] P 1f10f80f13c84e80b03eda7085a3b012: FlushMRSOp(683d4041910f4355910e188c0b7df6c7) complete. Timing: real 0.429s user 0.033s sys 0.015s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":7786,"dirs.run_cpu_time_us":218,"dirs.run_wall_time_us":2538,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":25519,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":18149,"threads_started":2,"wal-append.queue_time_us":288}
I20240503 02:14:29.299428 30312 maintenance_manager.cc:419] P 1f10f80f13c84e80b03eda7085a3b012: Scheduling UndoDeltaBlockGCOp(683d4041910f4355910e188c0b7df6c7): 4839 bytes on disk
I20240503 02:14:29.299899 30203 maintenance_manager.cc:643] P 1f10f80f13c84e80b03eda7085a3b012: UndoDeltaBlockGCOp(683d4041910f4355910e188c0b7df6c7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240503 02:14:29.432946 30132 raft_consensus.cc:1203] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:14:29.624415 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:29.664983 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:29.669260 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:29.695348 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:29.695775 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:29.714185 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:29.799547 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:29.819810 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:29.938249 434 consensus_queue.cc:237] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:29.940375 307 raft_consensus.cc:1261] T 2baf70e0e1894cea9fdeb92279bc0cff P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:29.940513 30133 raft_consensus.cc:1261] T 2baf70e0e1894cea9fdeb92279bc0cff P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:29.941071 13177 consensus_queue.cc:1035] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:29.941313 12861 consensus_queue.cc:1035] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:29.943938 12861 raft_consensus.cc:2932] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:29.944489 307 raft_consensus.cc:2932] T 2baf70e0e1894cea9fdeb92279bc0cff P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:29.947702 30133 raft_consensus.cc:2932] T 2baf70e0e1894cea9fdeb92279bc0cff P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:29.952170 29729 catalog_manager.cc:5484] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:29.967321 32619 tablet_service.cc:1514] Processing DeleteTablet for tablet 2baf70e0e1894cea9fdeb92279bc0cff with delete_type TABLET_DATA_TOMBSTONED (TS 9a4d92bc25374fb3960fde7d10be309d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:38200
I20240503 02:14:29.967953 13385 tablet_replica.cc:330] T 2baf70e0e1894cea9fdeb92279bc0cff P 9a4d92bc25374fb3960fde7d10be309d: stopping tablet replica
I20240503 02:14:29.968227 13385 raft_consensus.cc:2229] T 2baf70e0e1894cea9fdeb92279bc0cff P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:29.968423 13385 raft_consensus.cc:2258] T 2baf70e0e1894cea9fdeb92279bc0cff P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:29.969010 13385 ts_tablet_manager.cc:1875] T 2baf70e0e1894cea9fdeb92279bc0cff P 9a4d92bc25374fb3960fde7d10be309d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:29.974798 13385 ts_tablet_manager.cc:1888] T 2baf70e0e1894cea9fdeb92279bc0cff P 9a4d92bc25374fb3960fde7d10be309d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:14:29.974920 13385 log.cc:1196] T 2baf70e0e1894cea9fdeb92279bc0cff P 9a4d92bc25374fb3960fde7d10be309d: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal/wals/2baf70e0e1894cea9fdeb92279bc0cff
I20240503 02:14:29.975672 29719 catalog_manager.cc:4830] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): tablet 2baf70e0e1894cea9fdeb92279bc0cff (table rebalancer_extra_table_12 [id=ff2eb02eb88b44a0b0da2abc6307220a]) successfully deleted
I20240503 02:14:30.314522 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:30.354633 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:30.391299 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:30.397367 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:30.405323 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:30.414991 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:30.426882 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:30.442695 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:30.629470 434 consensus_queue.cc:237] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:30.631808 30133 raft_consensus.cc:1261] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:30.633589 13177 consensus_queue.cc:1035] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:30.634285 565 raft_consensus.cc:1261] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:30.635005 12861 consensus_queue.cc:1035] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:30.637451 12861 raft_consensus.cc:2932] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:30.642915 30133 raft_consensus.cc:2932] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:30.643218 565 raft_consensus.cc:2932] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:30.648285 29730 catalog_manager.cc:5484] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:30.675588 32619 tablet_service.cc:1514] Processing DeleteTablet for tablet 737a39a4f65b4e9ab2ce65c0d7f62dcb with delete_type TABLET_DATA_TOMBSTONED (TS 9a4d92bc25374fb3960fde7d10be309d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:38200
I20240503 02:14:30.688819 13475 tablet_replica.cc:330] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P 9a4d92bc25374fb3960fde7d10be309d: stopping tablet replica
I20240503 02:14:30.689122 13475 raft_consensus.cc:2229] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:30.689347 13475 raft_consensus.cc:2258] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:30.690039 13475 ts_tablet_manager.cc:1875] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P 9a4d92bc25374fb3960fde7d10be309d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:30.696302 13475 ts_tablet_manager.cc:1888] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P 9a4d92bc25374fb3960fde7d10be309d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:14:30.696537 13475 log.cc:1196] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P 9a4d92bc25374fb3960fde7d10be309d: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal/wals/737a39a4f65b4e9ab2ce65c0d7f62dcb
I20240503 02:14:30.697315 29719 catalog_manager.cc:4830] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): tablet 737a39a4f65b4e9ab2ce65c0d7f62dcb (table rebalancer_extra_table_0 [id=c9cc90e5637140278312ab93695a8afd]) successfully deleted
I20240503 02:14:30.921401 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:30.955828 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:30.980041 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:31.021250 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:31.024080 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:31.024470 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:31.025379 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:31.026134 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:31.228049 306 consensus_queue.cc:237] T 331905034afd48c19eb2bc2e2ec40280 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:14:31.230289 30004 raft_consensus.cc:1261] T 331905034afd48c19eb2bc2e2ec40280 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:31.231210 13288 consensus_queue.cc:1035] T 331905034afd48c19eb2bc2e2ec40280 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:31.231885 29874 raft_consensus.cc:1261] T 331905034afd48c19eb2bc2e2ec40280 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:31.232470 13288 consensus_queue.cc:1035] T 331905034afd48c19eb2bc2e2ec40280 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:31.236384 13288 raft_consensus.cc:2932] T 331905034afd48c19eb2bc2e2ec40280 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:31.240370 29874 raft_consensus.cc:2932] T 331905034afd48c19eb2bc2e2ec40280 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:31.240319 30004 raft_consensus.cc:2932] T 331905034afd48c19eb2bc2e2ec40280 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:31.247059 29727 catalog_manager.cc:5484] T 331905034afd48c19eb2bc2e2ec40280 P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:31.267231 32619 tablet_service.cc:1514] Processing DeleteTablet for tablet 331905034afd48c19eb2bc2e2ec40280 with delete_type TABLET_DATA_TOMBSTONED (TS 9a4d92bc25374fb3960fde7d10be309d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:38200
I20240503 02:14:31.267802 13564 tablet_replica.cc:330] T 331905034afd48c19eb2bc2e2ec40280 P 9a4d92bc25374fb3960fde7d10be309d: stopping tablet replica
I20240503 02:14:31.268024 13564 raft_consensus.cc:2229] T 331905034afd48c19eb2bc2e2ec40280 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:31.268301 13564 raft_consensus.cc:2258] T 331905034afd48c19eb2bc2e2ec40280 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:31.268872 13564 ts_tablet_manager.cc:1875] T 331905034afd48c19eb2bc2e2ec40280 P 9a4d92bc25374fb3960fde7d10be309d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:31.274857 13564 ts_tablet_manager.cc:1888] T 331905034afd48c19eb2bc2e2ec40280 P 9a4d92bc25374fb3960fde7d10be309d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:14:31.275048 13564 log.cc:1196] T 331905034afd48c19eb2bc2e2ec40280 P 9a4d92bc25374fb3960fde7d10be309d: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal/wals/331905034afd48c19eb2bc2e2ec40280
I20240503 02:14:31.275818 29719 catalog_manager.cc:4830] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): tablet 331905034afd48c19eb2bc2e2ec40280 (table rebalancer_extra_table_12 [id=ff2eb02eb88b44a0b0da2abc6307220a]) successfully deleted
I20240503 02:14:31.320500 565 consensus_queue.cc:237] T 4563386631774e74a4b12fc71714c1a0 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:31.323520 32640 raft_consensus.cc:1261] T 4563386631774e74a4b12fc71714c1a0 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:14:31.323958 501 consensus_peers.cc:485] T 4563386631774e74a4b12fc71714c1a0 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Couldn't send request to peer a9094240860e43bfb129193f3bfd5be6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4563386631774e74a4b12fc71714c1a0. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:31.324730 13171 consensus_queue.cc:1035] T 4563386631774e74a4b12fc71714c1a0 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:31.335829 30004 raft_consensus.cc:1261] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:31.336350 13587 consensus_queue.cc:1035] T 4563386631774e74a4b12fc71714c1a0 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:31.339062 13171 raft_consensus.cc:2932] T 4563386631774e74a4b12fc71714c1a0 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:31.339618 30004 raft_consensus.cc:2932] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:31.354852 29731 catalog_manager.cc:5484] T 4563386631774e74a4b12fc71714c1a0 P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New cstate: current_term: 1 leader_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:31.359462 32640 raft_consensus.cc:2932] T 4563386631774e74a4b12fc71714c1a0 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:31.609407 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:31.634073 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:31.647111 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:31.664912 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:31.692464 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:31.694485 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:31.706583 13648 ts_tablet_manager.cc:902] T 4563386631774e74a4b12fc71714c1a0 P a9094240860e43bfb129193f3bfd5be6: Initiating tablet copy from peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:14:31.721662 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:31.753412 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:31.789996 29921 maintenance_manager.cc:419] P 16440e365430405fb10366ab70e2b429: Scheduling FlushMRSOp(b7a2201f9605454ea12df4e3a7e5a63e): perf score=0.033337
I20240503 02:14:31.818416 30052 maintenance_manager.cc:419] P 155735c7f8b14997ac4c770aee2a9401: Scheduling FlushMRSOp(88b57085de3144cea3f64922ae5ccf55): perf score=0.033341
I20240503 02:14:31.827395 30312 maintenance_manager.cc:419] P 1f10f80f13c84e80b03eda7085a3b012: Scheduling FlushMRSOp(8fdbaf1df10b46d5a47326415d0cd0e9): perf score=0.033340
I20240503 02:14:31.863327 13648 tablet_copy_client.cc:308] T 4563386631774e74a4b12fc71714c1a0 P a9094240860e43bfb129193f3bfd5be6: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.8:42233
I20240503 02:14:31.864213 588 tablet_copy_service.cc:140] P f575ce92c0224eeb80b9fae5f6561fbe: Received BeginTabletCopySession request for tablet 4563386631774e74a4b12fc71714c1a0 from peer a9094240860e43bfb129193f3bfd5be6 ({username='slave'} at 127.22.108.3:57589)
I20240503 02:14:31.864390 588 tablet_copy_service.cc:161] P f575ce92c0224eeb80b9fae5f6561fbe: Beginning new tablet copy session on tablet 4563386631774e74a4b12fc71714c1a0 from peer a9094240860e43bfb129193f3bfd5be6 at {username='slave'} at 127.22.108.3:57589: session id = a9094240860e43bfb129193f3bfd5be6-4563386631774e74a4b12fc71714c1a0
I20240503 02:14:31.867337 588 tablet_copy_source_session.cc:205] T 4563386631774e74a4b12fc71714c1a0 P f575ce92c0224eeb80b9fae5f6561fbe: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:31.879458 30182 maintenance_manager.cc:419] P a9094240860e43bfb129193f3bfd5be6: Scheduling FlushMRSOp(8fdbaf1df10b46d5a47326415d0cd0e9): perf score=0.033361
I20240503 02:14:31.881117 13648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4563386631774e74a4b12fc71714c1a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:31.924379 13648 tablet_copy_client.cc:783] T 4563386631774e74a4b12fc71714c1a0 P a9094240860e43bfb129193f3bfd5be6: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:31.924644 13648 tablet_copy_client.cc:647] T 4563386631774e74a4b12fc71714c1a0 P a9094240860e43bfb129193f3bfd5be6: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:31.927313 13648 tablet_copy_client.cc:523] T 4563386631774e74a4b12fc71714c1a0 P a9094240860e43bfb129193f3bfd5be6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:31.943068 13648 tablet_bootstrap.cc:492] T 4563386631774e74a4b12fc71714c1a0 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:14:31.996088 29812 maintenance_manager.cc:643] P 16440e365430405fb10366ab70e2b429: FlushMRSOp(b7a2201f9605454ea12df4e3a7e5a63e) complete. Timing: real 0.206s user 0.028s sys 0.035s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":37995,"dirs.run_cpu_time_us":220,"dirs.run_wall_time_us":14390,"drs_written":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4,"lbm_write_time_us":895,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":76041,"threads_started":2,"wal-append.queue_time_us":6472}
I20240503 02:14:31.998275 13648 tablet_bootstrap.cc:492] T 4563386631774e74a4b12fc71714c1a0 P a9094240860e43bfb129193f3bfd5be6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:31.998682 13648 tablet_bootstrap.cc:492] T 4563386631774e74a4b12fc71714c1a0 P a9094240860e43bfb129193f3bfd5be6: Bootstrap complete.
I20240503 02:14:31.998907 13648 ts_tablet_manager.cc:1367] T 4563386631774e74a4b12fc71714c1a0 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.056s user 0.004s sys 0.005s
I20240503 02:14:31.999395 29921 maintenance_manager.cc:419] P 16440e365430405fb10366ab70e2b429: Scheduling UndoDeltaBlockGCOp(b7a2201f9605454ea12df4e3a7e5a63e): 4838 bytes on disk
I20240503 02:14:32.001389 13648 raft_consensus.cc:348] T 4563386631774e74a4b12fc71714c1a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:32.001626 13648 raft_consensus.cc:729] T 4563386631774e74a4b12fc71714c1a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:14:32.007779 29812 maintenance_manager.cc:643] P 16440e365430405fb10366ab70e2b429: UndoDeltaBlockGCOp(b7a2201f9605454ea12df4e3a7e5a63e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240503 02:14:32.008723 13648 consensus_queue.cc:260] T 4563386631774e74a4b12fc71714c1a0 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:32.011162 13648 ts_tablet_manager.cc:1398] T 4563386631774e74a4b12fc71714c1a0 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.012s user 0.002s sys 0.001s
I20240503 02:14:32.012102 29943 maintenance_manager.cc:643] P 155735c7f8b14997ac4c770aee2a9401: FlushMRSOp(88b57085de3144cea3f64922ae5ccf55) complete. Timing: real 0.193s user 0.069s sys 0.046s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":61588,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":6897,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1092,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":61862,"threads_started":2,"wal-append.queue_time_us":283}
I20240503 02:14:32.016889 588 tablet_copy_service.cc:342] P f575ce92c0224eeb80b9fae5f6561fbe: Request end of tablet copy session a9094240860e43bfb129193f3bfd5be6-4563386631774e74a4b12fc71714c1a0 received from {username='slave'} at 127.22.108.3:57589
I20240503 02:14:32.017073 588 tablet_copy_service.cc:433] P f575ce92c0224eeb80b9fae5f6561fbe: ending tablet copy session a9094240860e43bfb129193f3bfd5be6-4563386631774e74a4b12fc71714c1a0 on tablet 4563386631774e74a4b12fc71714c1a0 with peer a9094240860e43bfb129193f3bfd5be6
I20240503 02:14:32.019284 30052 maintenance_manager.cc:419] P 155735c7f8b14997ac4c770aee2a9401: Scheduling FlushMRSOp(b7a2201f9605454ea12df4e3a7e5a63e): perf score=0.033394
I20240503 02:14:32.021739 30203 maintenance_manager.cc:643] P 1f10f80f13c84e80b03eda7085a3b012: FlushMRSOp(8fdbaf1df10b46d5a47326415d0cd0e9) complete. Timing: real 0.194s user 0.039s sys 0.020s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":4175,"dirs.run_cpu_time_us":167,"dirs.run_wall_time_us":4637,"drs_written":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3231,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":36805,"threads_started":2,"wal-append.queue_time_us":6839}
I20240503 02:14:32.025106 30312 maintenance_manager.cc:419] P 1f10f80f13c84e80b03eda7085a3b012: Scheduling UndoDeltaBlockGCOp(8fdbaf1df10b46d5a47326415d0cd0e9): 4923 bytes on disk
I20240503 02:14:32.027662 30203 maintenance_manager.cc:643] P 1f10f80f13c84e80b03eda7085a3b012: UndoDeltaBlockGCOp(8fdbaf1df10b46d5a47326415d0cd0e9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240503 02:14:32.062438 30073 maintenance_manager.cc:643] P a9094240860e43bfb129193f3bfd5be6: FlushMRSOp(8fdbaf1df10b46d5a47326415d0cd0e9) complete. Timing: real 0.182s user 0.047s sys 0.010s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":23320,"dirs.run_cpu_time_us":293,"dirs.run_wall_time_us":2002,"drs_written":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4,"lbm_write_time_us":902,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":31440,"threads_started":2,"wal-append.queue_time_us":7893}
I20240503 02:14:32.067278 30182 maintenance_manager.cc:419] P a9094240860e43bfb129193f3bfd5be6: Scheduling FlushMRSOp(88b57085de3144cea3f64922ae5ccf55): perf score=0.033404
I20240503 02:14:32.146407 29943 maintenance_manager.cc:643] P 155735c7f8b14997ac4c770aee2a9401: FlushMRSOp(b7a2201f9605454ea12df4e3a7e5a63e) complete. Timing: real 0.125s user 0.047s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":59,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":1321,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":917,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":5997,"threads_started":1,"wal-append.queue_time_us":1404}
I20240503 02:14:32.149212 30052 maintenance_manager.cc:419] P 155735c7f8b14997ac4c770aee2a9401: Scheduling UndoDeltaBlockGCOp(b7a2201f9605454ea12df4e3a7e5a63e): 4838 bytes on disk
I20240503 02:14:32.149947 29943 maintenance_manager.cc:643] P 155735c7f8b14997ac4c770aee2a9401: UndoDeltaBlockGCOp(b7a2201f9605454ea12df4e3a7e5a63e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
I20240503 02:14:32.152695 30052 maintenance_manager.cc:419] P 155735c7f8b14997ac4c770aee2a9401: Scheduling UndoDeltaBlockGCOp(88b57085de3144cea3f64922ae5ccf55): 4582 bytes on disk
I20240503 02:14:32.153260 29943 maintenance_manager.cc:643] P 155735c7f8b14997ac4c770aee2a9401: UndoDeltaBlockGCOp(88b57085de3144cea3f64922ae5ccf55) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240503 02:14:32.157538 30132 raft_consensus.cc:1203] T 4563386631774e74a4b12fc71714c1a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:14:32.187525 30073 maintenance_manager.cc:643] P a9094240860e43bfb129193f3bfd5be6: FlushMRSOp(88b57085de3144cea3f64922ae5ccf55) complete. Timing: real 0.113s user 0.064s sys 0.014s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":453,"dirs.run_cpu_time_us":227,"dirs.run_wall_time_us":1569,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":830,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":15031,"threads_started":1,"wal-append.queue_time_us":15124}
I20240503 02:14:32.191175 30182 maintenance_manager.cc:419] P a9094240860e43bfb129193f3bfd5be6: Scheduling UndoDeltaBlockGCOp(8fdbaf1df10b46d5a47326415d0cd0e9): 4923 bytes on disk
I20240503 02:14:32.191738 30073 maintenance_manager.cc:643] P a9094240860e43bfb129193f3bfd5be6: UndoDeltaBlockGCOp(8fdbaf1df10b46d5a47326415d0cd0e9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240503 02:14:32.194852 30182 maintenance_manager.cc:419] P a9094240860e43bfb129193f3bfd5be6: Scheduling UndoDeltaBlockGCOp(88b57085de3144cea3f64922ae5ccf55): 4582 bytes on disk
I20240503 02:14:32.197469 30073 maintenance_manager.cc:643] P a9094240860e43bfb129193f3bfd5be6: UndoDeltaBlockGCOp(88b57085de3144cea3f64922ae5ccf55) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
I20240503 02:14:32.427595 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:32.441417 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:32.452577 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:32.543421 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:32.543999 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:32.558509 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:32.579864 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:32.594014 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:32.739462 30004 consensus_queue.cc:237] T a6505f95a1c547748fb5cb7a4918b129 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:32.741444 30132 raft_consensus.cc:1261] T a6505f95a1c547748fb5cb7a4918b129 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:32.743571 434 raft_consensus.cc:1261] T a6505f95a1c547748fb5cb7a4918b129 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:32.743762 13672 consensus_queue.cc:1035] T a6505f95a1c547748fb5cb7a4918b129 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:32.744122 13175 consensus_queue.cc:1035] T a6505f95a1c547748fb5cb7a4918b129 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:32.747859 13672 raft_consensus.cc:2932] T a6505f95a1c547748fb5cb7a4918b129 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:32.752058 30132 raft_consensus.cc:2932] T a6505f95a1c547748fb5cb7a4918b129 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:32.756551 29732 catalog_manager.cc:5484] T a6505f95a1c547748fb5cb7a4918b129 P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: config changed from index 2 to 3, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:32.762048 435 raft_consensus.cc:2932] T a6505f95a1c547748fb5cb7a4918b129 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:32.782384 32755 tablet_service.cc:1514] Processing DeleteTablet for tablet a6505f95a1c547748fb5cb7a4918b129 with delete_type TABLET_DATA_TOMBSTONED (TS 8a666e798fe8477d8dd1e51b385d5ead not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46360
I20240503 02:14:32.783028 13761 tablet_replica.cc:330] T a6505f95a1c547748fb5cb7a4918b129 P 8a666e798fe8477d8dd1e51b385d5ead: stopping tablet replica
I20240503 02:14:32.783304 13761 raft_consensus.cc:2229] T a6505f95a1c547748fb5cb7a4918b129 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:32.783576 13761 raft_consensus.cc:2258] T a6505f95a1c547748fb5cb7a4918b129 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:32.784154 13761 ts_tablet_manager.cc:1875] T a6505f95a1c547748fb5cb7a4918b129 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:32.790323 13761 ts_tablet_manager.cc:1888] T a6505f95a1c547748fb5cb7a4918b129 P 8a666e798fe8477d8dd1e51b385d5ead: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:14:32.790438 13761 log.cc:1196] T a6505f95a1c547748fb5cb7a4918b129 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal/wals/a6505f95a1c547748fb5cb7a4918b129
I20240503 02:14:32.791288 29719 catalog_manager.cc:4830] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): tablet a6505f95a1c547748fb5cb7a4918b129 (table rebalancer_extra_table_22 [id=fedaa5e192b44061a6bd08563b273009]) successfully deleted
I20240503 02:14:33.086241 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:33.092844 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:33.154805 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:33.158623 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:33.179903 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:33.192597 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:33.255983 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:33.270668 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 02:14:33.329336 30199 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.3:42679, user_credentials={real_user=slave}} blocked reactor thread for 44560us
W20240503 02:14:33.337389 30200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.2:37873, user_credentials={real_user=slave}} blocked reactor thread for 61063.8us
I20240503 02:14:33.439363 565 consensus_queue.cc:237] T 4563386631774e74a4b12fc71714c1a0 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:33.441457 30133 raft_consensus.cc:1261] T 4563386631774e74a4b12fc71714c1a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:33.441560 30006 raft_consensus.cc:1261] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:33.442014 13587 consensus_queue.cc:1035] T 4563386631774e74a4b12fc71714c1a0 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:33.442358 13720 consensus_queue.cc:1035] T 4563386631774e74a4b12fc71714c1a0 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:33.445293 13587 raft_consensus.cc:2932] T 4563386631774e74a4b12fc71714c1a0 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:33.445922 30133 raft_consensus.cc:2932] T 4563386631774e74a4b12fc71714c1a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:33.455811 29726 catalog_manager.cc:5484] T 4563386631774e74a4b12fc71714c1a0 P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New cstate: current_term: 1 leader_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:33.464686 30006 raft_consensus.cc:2932] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:33.490468 32619 tablet_service.cc:1514] Processing DeleteTablet for tablet 4563386631774e74a4b12fc71714c1a0 with delete_type TABLET_DATA_TOMBSTONED (TS 9a4d92bc25374fb3960fde7d10be309d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:38200
I20240503 02:14:33.495347 13863 tablet_replica.cc:330] T 4563386631774e74a4b12fc71714c1a0 P 9a4d92bc25374fb3960fde7d10be309d: stopping tablet replica
I20240503 02:14:33.495657 13863 raft_consensus.cc:2229] T 4563386631774e74a4b12fc71714c1a0 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:33.495919 13863 raft_consensus.cc:2258] T 4563386631774e74a4b12fc71714c1a0 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:33.496788 13863 ts_tablet_manager.cc:1875] T 4563386631774e74a4b12fc71714c1a0 P 9a4d92bc25374fb3960fde7d10be309d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:33.507771 13863 ts_tablet_manager.cc:1888] T 4563386631774e74a4b12fc71714c1a0 P 9a4d92bc25374fb3960fde7d10be309d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:14:33.508020 13863 log.cc:1196] T 4563386631774e74a4b12fc71714c1a0 P 9a4d92bc25374fb3960fde7d10be309d: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal/wals/4563386631774e74a4b12fc71714c1a0
I20240503 02:14:33.508998 29719 catalog_manager.cc:4830] TS 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): tablet 4563386631774e74a4b12fc71714c1a0 (table rebalancer_extra_table_36 [id=9f19e29c56ab440eb34df78f91d5bd29]) successfully deleted
I20240503 02:14:33.863816 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:33.897869 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:33.927529 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:33.936542 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:34.020855 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:34.035053 32619 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:34.061064 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:34.078860 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:34.287866 29921 maintenance_manager.cc:419] P 16440e365430405fb10366ab70e2b429: Scheduling FlushMRSOp(b6c9fd3cbbd5453897aa072600db569d): perf score=0.033349
I20240503 02:14:34.302024 30312 maintenance_manager.cc:419] P 1f10f80f13c84e80b03eda7085a3b012: Scheduling FlushMRSOp(2bf8c34ee16d4e0fbc6817d63b2ba52e): perf score=0.033348
I20240503 02:14:34.328069 434 consensus_queue.cc:237] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
W20240503 02:14:34.340911 371 consensus_peers.cc:485] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): Couldn't send request to peer 1f10f80f13c84e80b03eda7085a3b012. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 52a541b1d3db4228bc5a27af521f0b93. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:34.341200 567 raft_consensus.cc:1261] T 52a541b1d3db4228bc5a27af521f0b93 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:34.341878 12861 consensus_queue.cc:1035] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:34.342121 29874 raft_consensus.cc:1261] T 52a541b1d3db4228bc5a27af521f0b93 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:34.342978 12861 consensus_queue.cc:1035] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:34.347496 29812 maintenance_manager.cc:643] P 16440e365430405fb10366ab70e2b429: FlushMRSOp(b6c9fd3cbbd5453897aa072600db569d) complete. Timing: real 0.059s user 0.040s sys 0.010s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":359,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":8647,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":931,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":3315,"threads_started":2,"wal-append.queue_time_us":2967}
I20240503 02:14:34.350284 29921 maintenance_manager.cc:419] P 16440e365430405fb10366ab70e2b429: Scheduling UndoDeltaBlockGCOp(b6c9fd3cbbd5453897aa072600db569d): 4921 bytes on disk
I20240503 02:14:34.350862 29812 maintenance_manager.cc:643] P 16440e365430405fb10366ab70e2b429: UndoDeltaBlockGCOp(b6c9fd3cbbd5453897aa072600db569d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240503 02:14:34.352555 29921 maintenance_manager.cc:419] P 16440e365430405fb10366ab70e2b429: Scheduling FlushMRSOp(2bf8c34ee16d4e0fbc6817d63b2ba52e): perf score=0.033359
I20240503 02:14:34.354298 12861 raft_consensus.cc:2932] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:14:34.359268 30203 maintenance_manager.cc:643] P 1f10f80f13c84e80b03eda7085a3b012: FlushMRSOp(2bf8c34ee16d4e0fbc6817d63b2ba52e) complete. Timing: real 0.057s user 0.039s sys 0.014s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":6429,"dirs.run_cpu_time_us":218,"dirs.run_wall_time_us":2114,"drs_written":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4,"lbm_write_time_us":906,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":7580,"threads_started":2,"wal-append.queue_time_us":1113}
I20240503 02:14:34.359449 567 raft_consensus.cc:2932] T 52a541b1d3db4228bc5a27af521f0b93 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:14:34.361344 29874 raft_consensus.cc:2932] T 52a541b1d3db4228bc5a27af521f0b93 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:14:34.372705 29727 catalog_manager.cc:5484] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:14:34.389271 30312 maintenance_manager.cc:419] P 1f10f80f13c84e80b03eda7085a3b012: Scheduling UndoDeltaBlockGCOp(2bf8c34ee16d4e0fbc6817d63b2ba52e): 4836 bytes on disk
I20240503 02:14:34.389909 30203 maintenance_manager.cc:643] P 1f10f80f13c84e80b03eda7085a3b012: UndoDeltaBlockGCOp(2bf8c34ee16d4e0fbc6817d63b2ba52e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240503 02:14:34.436427 30052 maintenance_manager.cc:419] P 155735c7f8b14997ac4c770aee2a9401: Scheduling FlushMRSOp(0007679ffe8047d79dd8ac778fb6f1ce): perf score=0.033383
I20240503 02:14:34.447535 29812 maintenance_manager.cc:643] P 16440e365430405fb10366ab70e2b429: FlushMRSOp(2bf8c34ee16d4e0fbc6817d63b2ba52e) complete. Timing: real 0.095s user 0.064s sys 0.018s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":54,"dirs.run_cpu_time_us":159,"dirs.run_wall_time_us":1321,"drs_written":1,"lbm_read_time_us":132,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1077,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":16574,"threads_started":1,"wal-append.queue_time_us":16694}
I20240503 02:14:34.450408 29921 maintenance_manager.cc:419] P 16440e365430405fb10366ab70e2b429: Scheduling UndoDeltaBlockGCOp(2bf8c34ee16d4e0fbc6817d63b2ba52e): 4836 bytes on disk
I20240503 02:14:34.451432 29812 maintenance_manager.cc:643] P 16440e365430405fb10366ab70e2b429: UndoDeltaBlockGCOp(2bf8c34ee16d4e0fbc6817d63b2ba52e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20240503 02:14:34.478122 30182 maintenance_manager.cc:419] P a9094240860e43bfb129193f3bfd5be6: Scheduling FlushMRSOp(0007679ffe8047d79dd8ac778fb6f1ce): perf score=0.033402
I20240503 02:14:34.528671 29943 maintenance_manager.cc:643] P 155735c7f8b14997ac4c770aee2a9401: FlushMRSOp(0007679ffe8047d79dd8ac778fb6f1ce) complete. Timing: real 0.092s user 0.043s sys 0.012s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":346,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":2231,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":908,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":2451,"threads_started":2,"wal-append.queue_time_us":928}
I20240503 02:14:34.531867 30052 maintenance_manager.cc:419] P 155735c7f8b14997ac4c770aee2a9401: Scheduling UndoDeltaBlockGCOp(0007679ffe8047d79dd8ac778fb6f1ce): 4583 bytes on disk
I20240503 02:14:34.532703 29943 maintenance_manager.cc:643] P 155735c7f8b14997ac4c770aee2a9401: UndoDeltaBlockGCOp(0007679ffe8047d79dd8ac778fb6f1ce) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4}
I20240503 02:14:34.559511 30073 maintenance_manager.cc:643] P a9094240860e43bfb129193f3bfd5be6: FlushMRSOp(0007679ffe8047d79dd8ac778fb6f1ce) complete. Timing: real 0.081s user 0.041s sys 0.013s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":21368,"dirs.run_cpu_time_us":215,"dirs.run_wall_time_us":2949,"drs_written":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1318,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":19327,"threads_started":2,"wal-append.queue_time_us":362}
I20240503 02:14:34.563004 30182 maintenance_manager.cc:419] P a9094240860e43bfb129193f3bfd5be6: Scheduling UndoDeltaBlockGCOp(0007679ffe8047d79dd8ac778fb6f1ce): 4583 bytes on disk
I20240503 02:14:34.564318 30073 maintenance_manager.cc:643] P a9094240860e43bfb129193f3bfd5be6: UndoDeltaBlockGCOp(0007679ffe8047d79dd8ac778fb6f1ce) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
I20240503 02:14:34.568965 30182 maintenance_manager.cc:419] P a9094240860e43bfb129193f3bfd5be6: Scheduling FlushMRSOp(b6c9fd3cbbd5453897aa072600db569d): perf score=0.033419
I20240503 02:14:34.580405 30265 consensus_queue.cc:237] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
W20240503 02:14:34.586558 30199 consensus_peers.cc:485] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5faccb32e55d4cb689ff491fd20884e6. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:34.586778 29874 raft_consensus.cc:1261] T 5faccb32e55d4cb689ff491fd20884e6 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:34.587464 30006 raft_consensus.cc:1261] T 5faccb32e55d4cb689ff491fd20884e6 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:34.587679 13934 consensus_queue.cc:1035] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:34.588109 13055 consensus_queue.cc:1035] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:34.591889 13055 raft_consensus.cc:2932] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:34.592444 30006 raft_consensus.cc:2932] T 5faccb32e55d4cb689ff491fd20884e6 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:34.596433 29874 raft_consensus.cc:2932] T 5faccb32e55d4cb689ff491fd20884e6 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:34.604074 29733 catalog_manager.cc:5484] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: config changed from index -1 to 2, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:34.656626 30073 maintenance_manager.cc:643] P a9094240860e43bfb129193f3bfd5be6: FlushMRSOp(b6c9fd3cbbd5453897aa072600db569d) complete. Timing: real 0.087s user 0.060s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":68,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":1383,"drs_written":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":814,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":1720,"threads_started":1,"wal-append.queue_time_us":4523}
I20240503 02:14:34.659093 30182 maintenance_manager.cc:419] P a9094240860e43bfb129193f3bfd5be6: Scheduling UndoDeltaBlockGCOp(b6c9fd3cbbd5453897aa072600db569d): 4921 bytes on disk
I20240503 02:14:34.659888 30073 maintenance_manager.cc:643] P a9094240860e43bfb129193f3bfd5be6: UndoDeltaBlockGCOp(b6c9fd3cbbd5453897aa072600db569d) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":139,"lbm_reads_lt_1ms":4}
I20240503 02:14:34.686647 29874 consensus_queue.cc:237] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:14:34.689105 305 raft_consensus.cc:1261] T 32a2dbf80daf40d6b21442a39f62f14c P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:14:34.689569 30133 raft_consensus.cc:1261] T 32a2dbf80daf40d6b21442a39f62f14c P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:14:34.689993 13881 consensus_queue.cc:1035] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 02:14:34.690263 29809 consensus_peers.cc:485] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 -> Peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): Couldn't send request to peer fbbc2be6382e47908ecb7c68c367bd26. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32a2dbf80daf40d6b21442a39f62f14c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:34.690276 13733 consensus_queue.cc:1035] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:14:34.695508 13881 raft_consensus.cc:2932] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:14:34.696025 30133 raft_consensus.cc:2932] T 32a2dbf80daf40d6b21442a39f62f14c P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:14:34.701630 29732 catalog_manager.cc:5484] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 reported cstate change: config changed from index 5 to 6, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:14:34.702824 305 raft_consensus.cc:2932] T 32a2dbf80daf40d6b21442a39f62f14c P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:14:34.740725 30264 consensus_queue.cc:237] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:34.743947 29874 raft_consensus.cc:1261] T fa288463ae154e28afd9c71b4b55ef3a P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:14:34.744123 565 raft_consensus.cc:1261] T fa288463ae154e28afd9c71b4b55ef3a P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:14:34.744415 13934 consensus_queue.cc:1035] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 02:14:34.744616 30200 consensus_peers.cc:485] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): Couldn't send request to peer 8a666e798fe8477d8dd1e51b385d5ead. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fa288463ae154e28afd9c71b4b55ef3a. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:34.745114 13934 consensus_queue.cc:1035] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:14:34.748788 13835 raft_consensus.cc:2932] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:34.749426 565 raft_consensus.cc:2932] T fa288463ae154e28afd9c71b4b55ef3a P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:34.756516 29733 catalog_manager.cc:5484] T fa288463ae154e28afd9c71b4b55ef3a P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: config changed from index 5 to 6, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:34.757918 29873 raft_consensus.cc:2932] T fa288463ae154e28afd9c71b4b55ef3a P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:34.797572 30006 consensus_queue.cc:237] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:14:34.800122 435 raft_consensus.cc:1261] T 04e47da8cf7445dcacaf1f7c8950bd34 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240503 02:14:34.800318 29940 consensus_peers.cc:485] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04e47da8cf7445dcacaf1f7c8950bd34. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:34.800534 30265 raft_consensus.cc:1261] T 04e47da8cf7445dcacaf1f7c8950bd34 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:14:34.801746 13175 consensus_queue.cc:1035] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:14:34.802107 14046 consensus_queue.cc:1035] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:14:34.806615 14046 raft_consensus.cc:2932] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:34.809693 435 raft_consensus.cc:2932] T 04e47da8cf7445dcacaf1f7c8950bd34 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:34.815629 30265 raft_consensus.cc:2932] T 04e47da8cf7445dcacaf1f7c8950bd34 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:34.816493 29732 catalog_manager.cc:5484] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: config changed from index 5 to 6, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:34.831040 14051 ts_tablet_manager.cc:902] T 52a541b1d3db4228bc5a27af521f0b93 P 1f10f80f13c84e80b03eda7085a3b012: Initiating tablet copy from peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:14:34.842687 14051 tablet_copy_client.cc:308] T 52a541b1d3db4228bc5a27af521f0b93 P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.7:44241
I20240503 02:14:34.843425 457 tablet_copy_service.cc:140] P fbbc2be6382e47908ecb7c68c367bd26: Received BeginTabletCopySession request for tablet 52a541b1d3db4228bc5a27af521f0b93 from peer 1f10f80f13c84e80b03eda7085a3b012 ({username='slave'} at 127.22.108.4:42187)
I20240503 02:14:34.843566 457 tablet_copy_service.cc:161] P fbbc2be6382e47908ecb7c68c367bd26: Beginning new tablet copy session on tablet 52a541b1d3db4228bc5a27af521f0b93 from peer 1f10f80f13c84e80b03eda7085a3b012 at {username='slave'} at 127.22.108.4:42187: session id = 1f10f80f13c84e80b03eda7085a3b012-52a541b1d3db4228bc5a27af521f0b93
I20240503 02:14:34.846385 457 tablet_copy_source_session.cc:205] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:34.850196 14051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52a541b1d3db4228bc5a27af521f0b93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:34.857517 14051 tablet_copy_client.cc:783] T 52a541b1d3db4228bc5a27af521f0b93 P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:34.857836 14051 tablet_copy_client.cc:647] T 52a541b1d3db4228bc5a27af521f0b93 P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:34.859262 14051 tablet_copy_client.cc:523] T 52a541b1d3db4228bc5a27af521f0b93 P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:34.862285 14051 tablet_bootstrap.cc:492] T 52a541b1d3db4228bc5a27af521f0b93 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:14:34.867396 14051 tablet_bootstrap.cc:492] T 52a541b1d3db4228bc5a27af521f0b93 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:34.867767 14051 tablet_bootstrap.cc:492] T 52a541b1d3db4228bc5a27af521f0b93 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap complete.
I20240503 02:14:34.868017 14051 ts_tablet_manager.cc:1367] T 52a541b1d3db4228bc5a27af521f0b93 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20240503 02:14:34.868656 14051 raft_consensus.cc:348] T 52a541b1d3db4228bc5a27af521f0b93 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:14:34.868860 14051 raft_consensus.cc:729] T 52a541b1d3db4228bc5a27af521f0b93 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:14:34.869295 14051 consensus_queue.cc:260] T 52a541b1d3db4228bc5a27af521f0b93 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:14:34.869927 14051 ts_tablet_manager.cc:1398] T 52a541b1d3db4228bc5a27af521f0b93 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:14:34.871027 457 tablet_copy_service.cc:342] P fbbc2be6382e47908ecb7c68c367bd26: Request end of tablet copy session 1f10f80f13c84e80b03eda7085a3b012-52a541b1d3db4228bc5a27af521f0b93 received from {username='slave'} at 127.22.108.4:42187
I20240503 02:14:34.871135 457 tablet_copy_service.cc:433] P fbbc2be6382e47908ecb7c68c367bd26: ending tablet copy session 1f10f80f13c84e80b03eda7085a3b012-52a541b1d3db4228bc5a27af521f0b93 on tablet 52a541b1d3db4228bc5a27af521f0b93 with peer 1f10f80f13c84e80b03eda7085a3b012
I20240503 02:14:34.893541 305 consensus_queue.cc:237] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
W20240503 02:14:34.896080 32709 consensus_peers.cc:485] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Couldn't send request to peer 155735c7f8b14997ac4c770aee2a9401. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5cf58a111e3048449303ec45ac091b42. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:34.896582 29873 raft_consensus.cc:1261] T 5cf58a111e3048449303ec45ac091b42 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:34.896958 565 raft_consensus.cc:1261] T 5cf58a111e3048449303ec45ac091b42 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:34.897253 14072 consensus_queue.cc:1035] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:34.897572 13227 consensus_queue.cc:1035] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:34.911549 14072 raft_consensus.cc:2932] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:14:34.912149 567 raft_consensus.cc:2932] T 5cf58a111e3048449303ec45ac091b42 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:14:34.917490 29873 raft_consensus.cc:2932] T 5cf58a111e3048449303ec45ac091b42 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:14:34.921947 29732 catalog_manager.cc:5484] T 5cf58a111e3048449303ec45ac091b42 P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:14:34.965626 30006 consensus_queue.cc:237] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:34.967382 32641 raft_consensus.cc:1261] T 0007679ffe8047d79dd8ac778fb6f1ce P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:14:34.967432 30133 raft_consensus.cc:1261] T 0007679ffe8047d79dd8ac778fb6f1ce P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240503 02:14:34.967571 29940 consensus_peers.cc:485] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 -> Peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): Couldn't send request to peer 8a666e798fe8477d8dd1e51b385d5ead. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0007679ffe8047d79dd8ac778fb6f1ce. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:34.967886 14046 consensus_queue.cc:1035] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:14:34.968142 13935 consensus_queue.cc:1035] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:14:34.970978 13935 raft_consensus.cc:2932] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:34.973500 32641 raft_consensus.cc:2932] T 0007679ffe8047d79dd8ac778fb6f1ce P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:34.974823 30133 raft_consensus.cc:2932] T 0007679ffe8047d79dd8ac778fb6f1ce P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:34.978924 29727 catalog_manager.cc:5484] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: config changed from index 5 to 6, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:35.025768 30265 consensus_queue.cc:237] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:14:35.028127 305 raft_consensus.cc:1261] T 8f5e6ff35b994d46a1e1fe216eb356ce P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:35.028405 434 raft_consensus.cc:1261] T 8f5e6ff35b994d46a1e1fe216eb356ce P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:14:35.028491 30200 consensus_peers.cc:485] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f5e6ff35b994d46a1e1fe216eb356ce. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:35.028764 13835 consensus_queue.cc:1035] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:35.029083 13934 consensus_queue.cc:1035] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:35.034210 13835 raft_consensus.cc:2932] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:35.034904 305 raft_consensus.cc:2932] T 8f5e6ff35b994d46a1e1fe216eb356ce P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:35.038022 434 raft_consensus.cc:2932] T 8f5e6ff35b994d46a1e1fe216eb356ce P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:35.040838 29732 catalog_manager.cc:5484] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: config changed from index -1 to 2, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:35.053216 14110 ts_tablet_manager.cc:902] T 5faccb32e55d4cb689ff491fd20884e6 P f575ce92c0224eeb80b9fae5f6561fbe: Initiating tablet copy from peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:14:35.062170 14110 tablet_copy_client.cc:308] T 5faccb32e55d4cb689ff491fd20884e6 P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.4:33935
I20240503 02:14:35.062889 30286 tablet_copy_service.cc:140] P 1f10f80f13c84e80b03eda7085a3b012: Received BeginTabletCopySession request for tablet 5faccb32e55d4cb689ff491fd20884e6 from peer f575ce92c0224eeb80b9fae5f6561fbe ({username='slave'} at 127.22.108.8:51621)
I20240503 02:14:35.063104 30286 tablet_copy_service.cc:161] P 1f10f80f13c84e80b03eda7085a3b012: Beginning new tablet copy session on tablet 5faccb32e55d4cb689ff491fd20884e6 from peer f575ce92c0224eeb80b9fae5f6561fbe at {username='slave'} at 127.22.108.8:51621: session id = f575ce92c0224eeb80b9fae5f6561fbe-5faccb32e55d4cb689ff491fd20884e6
I20240503 02:14:35.065757 30286 tablet_copy_source_session.cc:205] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:35.067289 14110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5faccb32e55d4cb689ff491fd20884e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:35.069324 14120 ts_tablet_manager.cc:902] T 32a2dbf80daf40d6b21442a39f62f14c P fbbc2be6382e47908ecb7c68c367bd26: Initiating tablet copy from peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:14:35.074421 14110 tablet_copy_client.cc:783] T 5faccb32e55d4cb689ff491fd20884e6 P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:35.074640 14110 tablet_copy_client.cc:647] T 5faccb32e55d4cb689ff491fd20884e6 P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:35.076470 14110 tablet_copy_client.cc:523] T 5faccb32e55d4cb689ff491fd20884e6 P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:35.080698 14110 tablet_bootstrap.cc:492] T 5faccb32e55d4cb689ff491fd20884e6 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:14:35.097496 14110 tablet_bootstrap.cc:492] T 5faccb32e55d4cb689ff491fd20884e6 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:35.097875 14110 tablet_bootstrap.cc:492] T 5faccb32e55d4cb689ff491fd20884e6 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap complete.
I20240503 02:14:35.098111 14110 ts_tablet_manager.cc:1367] T 5faccb32e55d4cb689ff491fd20884e6 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.004s
I20240503 02:14:35.098754 14110 raft_consensus.cc:348] T 5faccb32e55d4cb689ff491fd20884e6 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:14:35.099331 14110 raft_consensus.cc:729] T 5faccb32e55d4cb689ff491fd20884e6 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:14:35.099812 14110 consensus_queue.cc:260] T 5faccb32e55d4cb689ff491fd20884e6 P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:14:35.102080 14110 ts_tablet_manager.cc:1398] T 5faccb32e55d4cb689ff491fd20884e6 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:14:35.104804 30286 tablet_copy_service.cc:342] P 1f10f80f13c84e80b03eda7085a3b012: Request end of tablet copy session f575ce92c0224eeb80b9fae5f6561fbe-5faccb32e55d4cb689ff491fd20884e6 received from {username='slave'} at 127.22.108.8:51621
I20240503 02:14:35.104918 30286 tablet_copy_service.cc:433] P 1f10f80f13c84e80b03eda7085a3b012: ending tablet copy session f575ce92c0224eeb80b9fae5f6561fbe-5faccb32e55d4cb689ff491fd20884e6 on tablet 5faccb32e55d4cb689ff491fd20884e6 with peer f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:14:35.105813 14120 tablet_copy_client.cc:308] T 32a2dbf80daf40d6b21442a39f62f14c P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.1:36719
I20240503 02:14:35.106988 29895 tablet_copy_service.cc:140] P 16440e365430405fb10366ab70e2b429: Received BeginTabletCopySession request for tablet 32a2dbf80daf40d6b21442a39f62f14c from peer fbbc2be6382e47908ecb7c68c367bd26 ({username='slave'} at 127.22.108.7:58281)
I20240503 02:14:35.107157 29895 tablet_copy_service.cc:161] P 16440e365430405fb10366ab70e2b429: Beginning new tablet copy session on tablet 32a2dbf80daf40d6b21442a39f62f14c from peer fbbc2be6382e47908ecb7c68c367bd26 at {username='slave'} at 127.22.108.7:58281: session id = fbbc2be6382e47908ecb7c68c367bd26-32a2dbf80daf40d6b21442a39f62f14c
I20240503 02:14:35.110119 29895 tablet_copy_source_session.cc:205] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:35.111367 14120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32a2dbf80daf40d6b21442a39f62f14c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:35.120900 14120 tablet_copy_client.cc:783] T 32a2dbf80daf40d6b21442a39f62f14c P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:35.121187 14120 tablet_copy_client.cc:647] T 32a2dbf80daf40d6b21442a39f62f14c P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:35.123077 14120 tablet_copy_client.cc:523] T 32a2dbf80daf40d6b21442a39f62f14c P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:35.128868 14120 tablet_bootstrap.cc:492] T 32a2dbf80daf40d6b21442a39f62f14c P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:14:35.135483 30006 consensus_queue.cc:237] T 8929e26a53274cecbb808dbd8b203a64 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:35.137862 29873 raft_consensus.cc:1261] T 8929e26a53274cecbb808dbd8b203a64 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:35.138559 305 raft_consensus.cc:1261] T 8929e26a53274cecbb808dbd8b203a64 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:14:35.138972 29939 consensus_peers.cc:485] T 8929e26a53274cecbb808dbd8b203a64 P 155735c7f8b14997ac4c770aee2a9401 -> Peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Couldn't send request to peer a9094240860e43bfb129193f3bfd5be6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8929e26a53274cecbb808dbd8b203a64. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:35.138974 14045 consensus_queue.cc:1035] T 8929e26a53274cecbb808dbd8b203a64 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:35.139261 14046 consensus_queue.cc:1035] T 8929e26a53274cecbb808dbd8b203a64 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:35.142736 14046 raft_consensus.cc:2932] T 8929e26a53274cecbb808dbd8b203a64 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:35.143604 29873 raft_consensus.cc:2932] T 8929e26a53274cecbb808dbd8b203a64 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:35.150799 29727 catalog_manager.cc:5484] T 8929e26a53274cecbb808dbd8b203a64 P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:35.152882 305 raft_consensus.cc:2932] T 8929e26a53274cecbb808dbd8b203a64 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:35.185441 14120 tablet_bootstrap.cc:492] T 32a2dbf80daf40d6b21442a39f62f14c P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:35.190557 14120 tablet_bootstrap.cc:492] T 32a2dbf80daf40d6b21442a39f62f14c P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap complete.
I20240503 02:14:35.190860 14120 ts_tablet_manager.cc:1367] T 32a2dbf80daf40d6b21442a39f62f14c P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.062s user 0.048s sys 0.000s
I20240503 02:14:35.192778 14120 raft_consensus.cc:348] T 32a2dbf80daf40d6b21442a39f62f14c P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:14:35.193048 14120 raft_consensus.cc:729] T 32a2dbf80daf40d6b21442a39f62f14c P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:14:35.193256 14120 consensus_queue.cc:260] T 32a2dbf80daf40d6b21442a39f62f14c P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:14:35.193779 14120 ts_tablet_manager.cc:1398] T 32a2dbf80daf40d6b21442a39f62f14c P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:14:35.194483 29895 tablet_copy_service.cc:342] P 16440e365430405fb10366ab70e2b429: Request end of tablet copy session fbbc2be6382e47908ecb7c68c367bd26-32a2dbf80daf40d6b21442a39f62f14c received from {username='slave'} at 127.22.108.7:58281
I20240503 02:14:35.194607 29895 tablet_copy_service.cc:433] P 16440e365430405fb10366ab70e2b429: ending tablet copy session fbbc2be6382e47908ecb7c68c367bd26-32a2dbf80daf40d6b21442a39f62f14c on tablet 32a2dbf80daf40d6b21442a39f62f14c with peer fbbc2be6382e47908ecb7c68c367bd26
I20240503 02:14:35.202587 14147 ts_tablet_manager.cc:902] T fa288463ae154e28afd9c71b4b55ef3a P 8a666e798fe8477d8dd1e51b385d5ead: Initiating tablet copy from peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:14:35.245767 14147 tablet_copy_client.cc:308] T fa288463ae154e28afd9c71b4b55ef3a P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.4:33935
I20240503 02:14:35.247320 30286 tablet_copy_service.cc:140] P 1f10f80f13c84e80b03eda7085a3b012: Received BeginTabletCopySession request for tablet fa288463ae154e28afd9c71b4b55ef3a from peer 8a666e798fe8477d8dd1e51b385d5ead ({username='slave'} at 127.22.108.6:54483)
I20240503 02:14:35.247473 30286 tablet_copy_service.cc:161] P 1f10f80f13c84e80b03eda7085a3b012: Beginning new tablet copy session on tablet fa288463ae154e28afd9c71b4b55ef3a from peer 8a666e798fe8477d8dd1e51b385d5ead at {username='slave'} at 127.22.108.6:54483: session id = 8a666e798fe8477d8dd1e51b385d5ead-fa288463ae154e28afd9c71b4b55ef3a
I20240503 02:14:35.250097 30286 tablet_copy_source_session.cc:205] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:35.251435 14147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa288463ae154e28afd9c71b4b55ef3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:35.256789 14147 tablet_copy_client.cc:783] T fa288463ae154e28afd9c71b4b55ef3a P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:35.257054 14147 tablet_copy_client.cc:647] T fa288463ae154e28afd9c71b4b55ef3a P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:35.263129 14147 tablet_copy_client.cc:523] T fa288463ae154e28afd9c71b4b55ef3a P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:35.266479 14147 tablet_bootstrap.cc:492] T fa288463ae154e28afd9c71b4b55ef3a P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:14:35.274425 32640 consensus_queue.cc:237] T 58e1aaecf47944569296fc1e0780efb5 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:14:35.277349 30265 raft_consensus.cc:1261] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20240503 02:14:35.277501 32573 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 9a4d92bc25374fb3960fde7d10be309d -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 58e1aaecf47944569296fc1e0780efb5. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:35.278007 13510 consensus_queue.cc:1035] T 58e1aaecf47944569296fc1e0780efb5 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:14:35.282096 434 raft_consensus.cc:1261] T 58e1aaecf47944569296fc1e0780efb5 P fbbc2be6382e47908ecb7c68c367bd26 [term 2 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:14:35.282932 13510 consensus_queue.cc:1035] T 58e1aaecf47944569296fc1e0780efb5 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:14:35.293687 13510 raft_consensus.cc:2932] T 58e1aaecf47944569296fc1e0780efb5 P 9a4d92bc25374fb3960fde7d10be309d [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:35.296792 30265 raft_consensus.cc:1203] T 52a541b1d3db4228bc5a27af521f0b93 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:14:35.310772 30265 raft_consensus.cc:2932] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:35.317215 434 raft_consensus.cc:2932] T 58e1aaecf47944569296fc1e0780efb5 P fbbc2be6382e47908ecb7c68c367bd26 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:35.324820 14177 ts_tablet_manager.cc:902] T 5cf58a111e3048449303ec45ac091b42 P 155735c7f8b14997ac4c770aee2a9401: Initiating tablet copy from peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:14:35.325726 29732 catalog_manager.cc:5484] T 58e1aaecf47944569296fc1e0780efb5 P 9a4d92bc25374fb3960fde7d10be309d reported cstate change: config changed from index 4 to 5, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New cstate: current_term: 2 leader_uuid: "9a4d92bc25374fb3960fde7d10be309d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:35.326506 14147 tablet_bootstrap.cc:492] T fa288463ae154e28afd9c71b4b55ef3a P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:35.326903 14147 tablet_bootstrap.cc:492] T fa288463ae154e28afd9c71b4b55ef3a P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap complete.
I20240503 02:14:35.327183 14147 ts_tablet_manager.cc:1367] T fa288463ae154e28afd9c71b4b55ef3a P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.061s user 0.045s sys 0.003s
I20240503 02:14:35.327864 14147 raft_consensus.cc:348] T fa288463ae154e28afd9c71b4b55ef3a P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:35.328164 14147 raft_consensus.cc:729] T fa288463ae154e28afd9c71b4b55ef3a P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:14:35.328377 14147 consensus_queue.cc:260] T fa288463ae154e28afd9c71b4b55ef3a P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:35.329449 14147 ts_tablet_manager.cc:1398] T fa288463ae154e28afd9c71b4b55ef3a P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:14:35.332839 30286 tablet_copy_service.cc:342] P 1f10f80f13c84e80b03eda7085a3b012: Request end of tablet copy session 8a666e798fe8477d8dd1e51b385d5ead-fa288463ae154e28afd9c71b4b55ef3a received from {username='slave'} at 127.22.108.6:54483
I20240503 02:14:35.332943 30286 tablet_copy_service.cc:433] P 1f10f80f13c84e80b03eda7085a3b012: ending tablet copy session 8a666e798fe8477d8dd1e51b385d5ead-fa288463ae154e28afd9c71b4b55ef3a on tablet fa288463ae154e28afd9c71b4b55ef3a with peer 8a666e798fe8477d8dd1e51b385d5ead
I20240503 02:14:35.343477 14177 tablet_copy_client.cc:308] T 5cf58a111e3048449303ec45ac091b42 P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.6:39975
I20240503 02:14:35.344244 327 tablet_copy_service.cc:140] P 8a666e798fe8477d8dd1e51b385d5ead: Received BeginTabletCopySession request for tablet 5cf58a111e3048449303ec45ac091b42 from peer 155735c7f8b14997ac4c770aee2a9401 ({username='slave'} at 127.22.108.2:53863)
I20240503 02:14:35.344399 327 tablet_copy_service.cc:161] P 8a666e798fe8477d8dd1e51b385d5ead: Beginning new tablet copy session on tablet 5cf58a111e3048449303ec45ac091b42 from peer 155735c7f8b14997ac4c770aee2a9401 at {username='slave'} at 127.22.108.2:53863: session id = 155735c7f8b14997ac4c770aee2a9401-5cf58a111e3048449303ec45ac091b42
I20240503 02:14:35.347311 327 tablet_copy_source_session.cc:205] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:35.348851 14177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cf58a111e3048449303ec45ac091b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:35.355254 14177 tablet_copy_client.cc:783] T 5cf58a111e3048449303ec45ac091b42 P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:35.355480 14177 tablet_copy_client.cc:647] T 5cf58a111e3048449303ec45ac091b42 P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:35.357271 14177 tablet_copy_client.cc:523] T 5cf58a111e3048449303ec45ac091b42 P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:35.360616 14177 tablet_bootstrap.cc:492] T 5cf58a111e3048449303ec45ac091b42 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
I20240503 02:14:35.366564 14177 tablet_bootstrap.cc:492] T 5cf58a111e3048449303ec45ac091b42 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:35.366938 14177 tablet_bootstrap.cc:492] T 5cf58a111e3048449303ec45ac091b42 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap complete.
I20240503 02:14:35.367187 14177 ts_tablet_manager.cc:1367] T 5cf58a111e3048449303ec45ac091b42 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240503 02:14:35.367899 14177 raft_consensus.cc:348] T 5cf58a111e3048449303ec45ac091b42 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:14:35.368113 14177 raft_consensus.cc:729] T 5cf58a111e3048449303ec45ac091b42 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:14:35.368490 14177 consensus_queue.cc:260] T 5cf58a111e3048449303ec45ac091b42 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:14:35.369366 14177 ts_tablet_manager.cc:1398] T 5cf58a111e3048449303ec45ac091b42 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:14:35.370913 327 tablet_copy_service.cc:342] P 8a666e798fe8477d8dd1e51b385d5ead: Request end of tablet copy session 155735c7f8b14997ac4c770aee2a9401-5cf58a111e3048449303ec45ac091b42 received from {username='slave'} at 127.22.108.2:53863
I20240503 02:14:35.371033 327 tablet_copy_service.cc:433] P 8a666e798fe8477d8dd1e51b385d5ead: ending tablet copy session 155735c7f8b14997ac4c770aee2a9401-5cf58a111e3048449303ec45ac091b42 on tablet 5cf58a111e3048449303ec45ac091b42 with peer 155735c7f8b14997ac4c770aee2a9401
I20240503 02:14:35.373692 14187 ts_tablet_manager.cc:902] T 04e47da8cf7445dcacaf1f7c8950bd34 P 9a4d92bc25374fb3960fde7d10be309d: Initiating tablet copy from peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:14:35.391837 30133 consensus_queue.cc:237] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
W20240503 02:14:35.396397 30070 consensus_peers.cc:485] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 -> Peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): Couldn't send request to peer fbbc2be6382e47908ecb7c68c367bd26. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 88b57085de3144cea3f64922ae5ccf55. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:35.396435 565 raft_consensus.cc:1261] T 88b57085de3144cea3f64922ae5ccf55 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:14:35.397161 30004 raft_consensus.cc:1261] T 88b57085de3144cea3f64922ae5ccf55 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:14:35.397461 13556 consensus_queue.cc:1035] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:14:35.397745 14132 consensus_queue.cc:1035] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:14:35.404989 13556 raft_consensus.cc:2932] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:14:35.406862 565 raft_consensus.cc:2932] T 88b57085de3144cea3f64922ae5ccf55 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:14:35.419911 29727 catalog_manager.cc:5484] T 88b57085de3144cea3f64922ae5ccf55 P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: config changed from index 5 to 6, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:14:35.425546 30004 raft_consensus.cc:2932] T 88b57085de3144cea3f64922ae5ccf55 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:14:35.439198 14203 ts_tablet_manager.cc:902] T 8f5e6ff35b994d46a1e1fe216eb356ce P 9a4d92bc25374fb3960fde7d10be309d: Initiating tablet copy from peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:14:35.439412 14187 tablet_copy_client.cc:308] T 04e47da8cf7445dcacaf1f7c8950bd34 P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.2:37873
I20240503 02:14:35.442256 30026 tablet_copy_service.cc:140] P 155735c7f8b14997ac4c770aee2a9401: Received BeginTabletCopySession request for tablet 04e47da8cf7445dcacaf1f7c8950bd34 from peer 9a4d92bc25374fb3960fde7d10be309d ({username='slave'} at 127.22.108.5:55617)
I20240503 02:14:35.442394 30026 tablet_copy_service.cc:161] P 155735c7f8b14997ac4c770aee2a9401: Beginning new tablet copy session on tablet 04e47da8cf7445dcacaf1f7c8950bd34 from peer 9a4d92bc25374fb3960fde7d10be309d at {username='slave'} at 127.22.108.5:55617: session id = 9a4d92bc25374fb3960fde7d10be309d-04e47da8cf7445dcacaf1f7c8950bd34
I20240503 02:14:35.445405 30026 tablet_copy_source_session.cc:205] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:35.451382 14187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04e47da8cf7445dcacaf1f7c8950bd34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:35.469172 14147 ts_tablet_manager.cc:902] T 0007679ffe8047d79dd8ac778fb6f1ce P 8a666e798fe8477d8dd1e51b385d5ead: Initiating tablet copy from peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:14:35.469484 14187 tablet_copy_client.cc:783] T 04e47da8cf7445dcacaf1f7c8950bd34 P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:35.469863 14187 tablet_copy_client.cc:647] T 04e47da8cf7445dcacaf1f7c8950bd34 P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:35.472002 14187 tablet_copy_client.cc:523] T 04e47da8cf7445dcacaf1f7c8950bd34 P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:35.476225 14187 tablet_bootstrap.cc:492] T 04e47da8cf7445dcacaf1f7c8950bd34 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:14:35.490154 565 raft_consensus.cc:1203] T 5faccb32e55d4cb689ff491fd20884e6 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:14:35.496925 14203 tablet_copy_client.cc:308] T 8f5e6ff35b994d46a1e1fe216eb356ce P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.4:33935
I20240503 02:14:35.497745 30286 tablet_copy_service.cc:140] P 1f10f80f13c84e80b03eda7085a3b012: Received BeginTabletCopySession request for tablet 8f5e6ff35b994d46a1e1fe216eb356ce from peer 9a4d92bc25374fb3960fde7d10be309d ({username='slave'} at 127.22.108.5:44575)
I20240503 02:14:35.497895 30286 tablet_copy_service.cc:161] P 1f10f80f13c84e80b03eda7085a3b012: Beginning new tablet copy session on tablet 8f5e6ff35b994d46a1e1fe216eb356ce from peer 9a4d92bc25374fb3960fde7d10be309d at {username='slave'} at 127.22.108.5:44575: session id = 9a4d92bc25374fb3960fde7d10be309d-8f5e6ff35b994d46a1e1fe216eb356ce
I20240503 02:14:35.506680 30286 tablet_copy_source_session.cc:205] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:35.508132 14203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f5e6ff35b994d46a1e1fe216eb356ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:35.520581 14203 tablet_copy_client.cc:783] T 8f5e6ff35b994d46a1e1fe216eb356ce P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:35.520857 14203 tablet_copy_client.cc:647] T 8f5e6ff35b994d46a1e1fe216eb356ce P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:35.522505 14203 tablet_copy_client.cc:523] T 8f5e6ff35b994d46a1e1fe216eb356ce P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:35.527150 14203 tablet_bootstrap.cc:492] T 8f5e6ff35b994d46a1e1fe216eb356ce P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:14:35.538728 14147 tablet_copy_client.cc:308] T 0007679ffe8047d79dd8ac778fb6f1ce P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.2:37873
I20240503 02:14:35.539718 30026 tablet_copy_service.cc:140] P 155735c7f8b14997ac4c770aee2a9401: Received BeginTabletCopySession request for tablet 0007679ffe8047d79dd8ac778fb6f1ce from peer 8a666e798fe8477d8dd1e51b385d5ead ({username='slave'} at 127.22.108.6:33785)
I20240503 02:14:35.539961 30026 tablet_copy_service.cc:161] P 155735c7f8b14997ac4c770aee2a9401: Beginning new tablet copy session on tablet 0007679ffe8047d79dd8ac778fb6f1ce from peer 8a666e798fe8477d8dd1e51b385d5ead at {username='slave'} at 127.22.108.6:33785: session id = 8a666e798fe8477d8dd1e51b385d5ead-0007679ffe8047d79dd8ac778fb6f1ce
I20240503 02:14:35.547977 30026 tablet_copy_source_session.cc:205] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:14:35.550472 14147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0007679ffe8047d79dd8ac778fb6f1ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:35.553160 14203 tablet_bootstrap.cc:492] T 8f5e6ff35b994d46a1e1fe216eb356ce P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:35.553671 14203 tablet_bootstrap.cc:492] T 8f5e6ff35b994d46a1e1fe216eb356ce P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap complete.
I20240503 02:14:35.553929 14203 ts_tablet_manager.cc:1367] T 8f5e6ff35b994d46a1e1fe216eb356ce P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.011s
I20240503 02:14:35.554663 14203 raft_consensus.cc:348] T 8f5e6ff35b994d46a1e1fe216eb356ce P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:14:35.554904 14203 raft_consensus.cc:729] T 8f5e6ff35b994d46a1e1fe216eb356ce P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:14:35.555173 14203 consensus_queue.cc:260] T 8f5e6ff35b994d46a1e1fe216eb356ce P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:14:35.557178 14203 ts_tablet_manager.cc:1398] T 8f5e6ff35b994d46a1e1fe216eb356ce P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:14:35.557727 30286 tablet_copy_service.cc:342] P 1f10f80f13c84e80b03eda7085a3b012: Request end of tablet copy session 9a4d92bc25374fb3960fde7d10be309d-8f5e6ff35b994d46a1e1fe216eb356ce received from {username='slave'} at 127.22.108.5:44575
I20240503 02:14:35.557832 30286 tablet_copy_service.cc:433] P 1f10f80f13c84e80b03eda7085a3b012: ending tablet copy session 9a4d92bc25374fb3960fde7d10be309d-8f5e6ff35b994d46a1e1fe216eb356ce on tablet 8f5e6ff35b994d46a1e1fe216eb356ce with peer 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:14:35.562188 14147 tablet_copy_client.cc:783] T 0007679ffe8047d79dd8ac778fb6f1ce P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Starting download of 5 data blocks...
I20240503 02:14:35.580611 14147 tablet_copy_client.cc:647] T 0007679ffe8047d79dd8ac778fb6f1ce P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:35.583933 14147 tablet_copy_client.cc:523] T 0007679ffe8047d79dd8ac778fb6f1ce P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:35.584955 14187 tablet_bootstrap.cc:492] T 04e47da8cf7445dcacaf1f7c8950bd34 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:35.585384 14187 tablet_bootstrap.cc:492] T 04e47da8cf7445dcacaf1f7c8950bd34 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap complete.
I20240503 02:14:35.585709 14187 ts_tablet_manager.cc:1367] T 04e47da8cf7445dcacaf1f7c8950bd34 P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.110s user 0.063s sys 0.011s
I20240503 02:14:35.586547 14187 raft_consensus.cc:348] T 04e47da8cf7445dcacaf1f7c8950bd34 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:14:35.586750 14187 raft_consensus.cc:729] T 04e47da8cf7445dcacaf1f7c8950bd34 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:14:35.586966 14187 consensus_queue.cc:260] T 04e47da8cf7445dcacaf1f7c8950bd34 P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:14:35.587918 14147 tablet_bootstrap.cc:492] T 0007679ffe8047d79dd8ac778fb6f1ce P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:14:35.588639 14187 ts_tablet_manager.cc:1398] T 04e47da8cf7445dcacaf1f7c8950bd34 P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:14:35.589277 30026 tablet_copy_service.cc:342] P 155735c7f8b14997ac4c770aee2a9401: Request end of tablet copy session 9a4d92bc25374fb3960fde7d10be309d-04e47da8cf7445dcacaf1f7c8950bd34 received from {username='slave'} at 127.22.108.5:55617
I20240503 02:14:35.589404 30026 tablet_copy_service.cc:433] P 155735c7f8b14997ac4c770aee2a9401: ending tablet copy session 9a4d92bc25374fb3960fde7d10be309d-04e47da8cf7445dcacaf1f7c8950bd34 on tablet 04e47da8cf7445dcacaf1f7c8950bd34 with peer 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:14:35.599061 14226 ts_tablet_manager.cc:902] T 8929e26a53274cecbb808dbd8b203a64 P a9094240860e43bfb129193f3bfd5be6: Initiating tablet copy from peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:14:35.614796 14147 tablet_bootstrap.cc:492] T 0007679ffe8047d79dd8ac778fb6f1ce P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:35.615253 14147 tablet_bootstrap.cc:492] T 0007679ffe8047d79dd8ac778fb6f1ce P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap complete.
I20240503 02:14:35.615571 14147 ts_tablet_manager.cc:1367] T 0007679ffe8047d79dd8ac778fb6f1ce P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.028s user 0.021s sys 0.005s
I20240503 02:14:35.616210 14147 raft_consensus.cc:348] T 0007679ffe8047d79dd8ac778fb6f1ce P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:35.616403 14147 raft_consensus.cc:729] T 0007679ffe8047d79dd8ac778fb6f1ce P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:14:35.616650 14147 consensus_queue.cc:260] T 0007679ffe8047d79dd8ac778fb6f1ce P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:35.617300 14147 ts_tablet_manager.cc:1398] T 0007679ffe8047d79dd8ac778fb6f1ce P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 02:14:35.618264 30026 tablet_copy_service.cc:342] P 155735c7f8b14997ac4c770aee2a9401: Request end of tablet copy session 8a666e798fe8477d8dd1e51b385d5ead-0007679ffe8047d79dd8ac778fb6f1ce received from {username='slave'} at 127.22.108.6:33785
I20240503 02:14:35.618424 30026 tablet_copy_service.cc:433] P 155735c7f8b14997ac4c770aee2a9401: ending tablet copy session 8a666e798fe8477d8dd1e51b385d5ead-0007679ffe8047d79dd8ac778fb6f1ce on tablet 0007679ffe8047d79dd8ac778fb6f1ce with peer 8a666e798fe8477d8dd1e51b385d5ead
I20240503 02:14:35.644956 14226 tablet_copy_client.cc:308] T 8929e26a53274cecbb808dbd8b203a64 P a9094240860e43bfb129193f3bfd5be6: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.2:37873
I20240503 02:14:35.648190 30026 tablet_copy_service.cc:140] P 155735c7f8b14997ac4c770aee2a9401: Received BeginTabletCopySession request for tablet 8929e26a53274cecbb808dbd8b203a64 from peer a9094240860e43bfb129193f3bfd5be6 ({username='slave'} at 127.22.108.3:41369)
I20240503 02:14:35.648373 30026 tablet_copy_service.cc:161] P 155735c7f8b14997ac4c770aee2a9401: Beginning new tablet copy session on tablet 8929e26a53274cecbb808dbd8b203a64 from peer a9094240860e43bfb129193f3bfd5be6 at {username='slave'} at 127.22.108.3:41369: session id = a9094240860e43bfb129193f3bfd5be6-8929e26a53274cecbb808dbd8b203a64
I20240503 02:14:35.654443 30135 consensus_queue.cc:237] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:35.660509 30026 tablet_copy_source_session.cc:205] T 8929e26a53274cecbb808dbd8b203a64 P 155735c7f8b14997ac4c770aee2a9401: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:35.662608 32641 raft_consensus.cc:1261] T e80b43971dc246d9bfd5e1c603f8415c P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:14:35.662776 29874 raft_consensus.cc:1261] T e80b43971dc246d9bfd5e1c603f8415c P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240503 02:14:35.663017 30070 consensus_peers.cc:485] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 -> Peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): Couldn't send request to peer 8a666e798fe8477d8dd1e51b385d5ead. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e80b43971dc246d9bfd5e1c603f8415c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:35.663303 14200 consensus_queue.cc:1035] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:14:35.663668 14202 consensus_queue.cc:1035] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:14:35.666504 14226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8929e26a53274cecbb808dbd8b203a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:35.672724 14226 tablet_copy_client.cc:783] T 8929e26a53274cecbb808dbd8b203a64 P a9094240860e43bfb129193f3bfd5be6: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:35.672998 14226 tablet_copy_client.cc:647] T 8929e26a53274cecbb808dbd8b203a64 P a9094240860e43bfb129193f3bfd5be6: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:35.674077 434 raft_consensus.cc:1203] T 32a2dbf80daf40d6b21442a39f62f14c P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 02:14:35.677008 14202 raft_consensus.cc:2932] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:35.680536 32640 raft_consensus.cc:2932] T e80b43971dc246d9bfd5e1c603f8415c P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:35.684590 14226 tablet_copy_client.cc:523] T 8929e26a53274cecbb808dbd8b203a64 P a9094240860e43bfb129193f3bfd5be6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:35.685873 306 raft_consensus.cc:1203] T fa288463ae154e28afd9c71b4b55ef3a P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 02:14:35.687983 14226 tablet_bootstrap.cc:492] T 8929e26a53274cecbb808dbd8b203a64 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
I20240503 02:14:35.688211 29733 catalog_manager.cc:5484] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 reported cstate change: config changed from index 5 to 6, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:35.698529 29873 raft_consensus.cc:2932] T e80b43971dc246d9bfd5e1c603f8415c P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:35.704242 14226 tablet_bootstrap.cc:492] T 8929e26a53274cecbb808dbd8b203a64 P a9094240860e43bfb129193f3bfd5be6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:35.704594 14226 tablet_bootstrap.cc:492] T 8929e26a53274cecbb808dbd8b203a64 P a9094240860e43bfb129193f3bfd5be6: Bootstrap complete.
I20240503 02:14:35.704825 14226 ts_tablet_manager.cc:1367] T 8929e26a53274cecbb808dbd8b203a64 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.004s
I20240503 02:14:35.705562 14226 raft_consensus.cc:348] T 8929e26a53274cecbb808dbd8b203a64 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:35.705835 14226 raft_consensus.cc:729] T 8929e26a53274cecbb808dbd8b203a64 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:14:35.706125 14226 consensus_queue.cc:260] T 8929e26a53274cecbb808dbd8b203a64 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:35.707810 14240 ts_tablet_manager.cc:902] T 58e1aaecf47944569296fc1e0780efb5 P f575ce92c0224eeb80b9fae5f6561fbe: Initiating tablet copy from peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:14:35.708971 14226 ts_tablet_manager.cc:1398] T 8929e26a53274cecbb808dbd8b203a64 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 02:14:35.710552 30026 tablet_copy_service.cc:342] P 155735c7f8b14997ac4c770aee2a9401: Request end of tablet copy session a9094240860e43bfb129193f3bfd5be6-8929e26a53274cecbb808dbd8b203a64 received from {username='slave'} at 127.22.108.3:41369
I20240503 02:14:35.710846 30026 tablet_copy_service.cc:433] P 155735c7f8b14997ac4c770aee2a9401: ending tablet copy session a9094240860e43bfb129193f3bfd5be6-8929e26a53274cecbb808dbd8b203a64 on tablet 8929e26a53274cecbb808dbd8b203a64 with peer a9094240860e43bfb129193f3bfd5be6
I20240503 02:14:35.717741 14240 tablet_copy_client.cc:308] T 58e1aaecf47944569296fc1e0780efb5 P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.5:35595
I20240503 02:14:35.718798 32662 tablet_copy_service.cc:140] P 9a4d92bc25374fb3960fde7d10be309d: Received BeginTabletCopySession request for tablet 58e1aaecf47944569296fc1e0780efb5 from peer f575ce92c0224eeb80b9fae5f6561fbe ({username='slave'} at 127.22.108.8:37451)
I20240503 02:14:35.718966 32662 tablet_copy_service.cc:161] P 9a4d92bc25374fb3960fde7d10be309d: Beginning new tablet copy session on tablet 58e1aaecf47944569296fc1e0780efb5 from peer f575ce92c0224eeb80b9fae5f6561fbe at {username='slave'} at 127.22.108.8:37451: session id = f575ce92c0224eeb80b9fae5f6561fbe-58e1aaecf47944569296fc1e0780efb5
I20240503 02:14:35.721421 32662 tablet_copy_source_session.cc:205] T 58e1aaecf47944569296fc1e0780efb5 P 9a4d92bc25374fb3960fde7d10be309d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:35.722856 14240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58e1aaecf47944569296fc1e0780efb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:35.728155 14240 tablet_copy_client.cc:783] T 58e1aaecf47944569296fc1e0780efb5 P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:35.728343 14240 tablet_copy_client.cc:647] T 58e1aaecf47944569296fc1e0780efb5 P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:35.730281 14240 tablet_copy_client.cc:523] T 58e1aaecf47944569296fc1e0780efb5 P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:35.733613 14240 tablet_bootstrap.cc:492] T 58e1aaecf47944569296fc1e0780efb5 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:14:35.735172 32640 raft_consensus.cc:1203] T 04e47da8cf7445dcacaf1f7c8950bd34 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 02:14:35.742311 14240 tablet_bootstrap.cc:492] T 58e1aaecf47944569296fc1e0780efb5 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:35.742630 14240 tablet_bootstrap.cc:492] T 58e1aaecf47944569296fc1e0780efb5 P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap complete.
I20240503 02:14:35.742889 14240 ts_tablet_manager.cc:1367] T 58e1aaecf47944569296fc1e0780efb5 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240503 02:14:35.743615 14240 raft_consensus.cc:348] T 58e1aaecf47944569296fc1e0780efb5 P f575ce92c0224eeb80b9fae5f6561fbe [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:14:35.743845 14240 raft_consensus.cc:729] T 58e1aaecf47944569296fc1e0780efb5 P f575ce92c0224eeb80b9fae5f6561fbe [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:14:35.744112 14240 consensus_queue.cc:260] T 58e1aaecf47944569296fc1e0780efb5 P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:14:35.744616 14240 ts_tablet_manager.cc:1398] T 58e1aaecf47944569296fc1e0780efb5 P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:14:35.745460 32662 tablet_copy_service.cc:342] P 9a4d92bc25374fb3960fde7d10be309d: Request end of tablet copy session f575ce92c0224eeb80b9fae5f6561fbe-58e1aaecf47944569296fc1e0780efb5 received from {username='slave'} at 127.22.108.8:37451
I20240503 02:14:35.745584 32662 tablet_copy_service.cc:433] P 9a4d92bc25374fb3960fde7d10be309d: ending tablet copy session f575ce92c0224eeb80b9fae5f6561fbe-58e1aaecf47944569296fc1e0780efb5 on tablet 58e1aaecf47944569296fc1e0780efb5 with peer f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:14:35.766188 30265 consensus_queue.cc:237] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:14:35.768319 30135 raft_consensus.cc:1261] T b90b887127f44c0d920225be10077c64 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:35.768610 434 raft_consensus.cc:1261] T b90b887127f44c0d920225be10077c64 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:14:35.768929 30200 consensus_peers.cc:485] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b90b887127f44c0d920225be10077c64. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:35.768975 13055 consensus_queue.cc:1035] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:35.769279 14027 consensus_queue.cc:1035] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:35.774171 13055 raft_consensus.cc:2932] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:35.774649 434 raft_consensus.cc:2932] T b90b887127f44c0d920225be10077c64 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:35.780064 30135 raft_consensus.cc:2932] T b90b887127f44c0d920225be10077c64 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:35.781519 29727 catalog_manager.cc:5484] T b90b887127f44c0d920225be10077c64 P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: config changed from index -1 to 2, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:35.795512 30004 raft_consensus.cc:1203] T 5cf58a111e3048449303ec45ac091b42 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:14:35.800733 353 maintenance_manager.cc:419] P 8a666e798fe8477d8dd1e51b385d5ead: Scheduling UndoDeltaBlockGCOp(0007679ffe8047d79dd8ac778fb6f1ce): 4583 bytes on disk
I20240503 02:14:35.802556 32712 maintenance_manager.cc:643] P 8a666e798fe8477d8dd1e51b385d5ead: UndoDeltaBlockGCOp(0007679ffe8047d79dd8ac778fb6f1ce) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240503 02:14:35.810564 14274 ts_tablet_manager.cc:902] T 88b57085de3144cea3f64922ae5ccf55 P fbbc2be6382e47908ecb7c68c367bd26: Initiating tablet copy from peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:14:35.815858 14274 tablet_copy_client.cc:308] T 88b57085de3144cea3f64922ae5ccf55 P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.3:42679
I20240503 02:14:35.816784 30156 tablet_copy_service.cc:140] P a9094240860e43bfb129193f3bfd5be6: Received BeginTabletCopySession request for tablet 88b57085de3144cea3f64922ae5ccf55 from peer fbbc2be6382e47908ecb7c68c367bd26 ({username='slave'} at 127.22.108.7:37671)
I20240503 02:14:35.816960 30156 tablet_copy_service.cc:161] P a9094240860e43bfb129193f3bfd5be6: Beginning new tablet copy session on tablet 88b57085de3144cea3f64922ae5ccf55 from peer fbbc2be6382e47908ecb7c68c367bd26 at {username='slave'} at 127.22.108.7:37671: session id = fbbc2be6382e47908ecb7c68c367bd26-88b57085de3144cea3f64922ae5ccf55
I20240503 02:14:35.820281 30156 tablet_copy_source_session.cc:205] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:14:35.821352 32640 raft_consensus.cc:1203] T 8f5e6ff35b994d46a1e1fe216eb356ce P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:14:35.823316 14274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88b57085de3144cea3f64922ae5ccf55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:35.831698 14274 tablet_copy_client.cc:783] T 88b57085de3144cea3f64922ae5ccf55 P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Starting download of 5 data blocks...
I20240503 02:14:35.841006 14274 tablet_copy_client.cc:647] T 88b57085de3144cea3f64922ae5ccf55 P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:35.844015 14274 tablet_copy_client.cc:523] T 88b57085de3144cea3f64922ae5ccf55 P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:35.849541 14274 tablet_bootstrap.cc:492] T 88b57085de3144cea3f64922ae5ccf55 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:14:35.860407 434 consensus_queue.cc:237] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:14:35.864056 306 raft_consensus.cc:1261] T b7b284cffcb04ce883a24f6a8796748f P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:35.864806 13959 consensus_queue.cc:1035] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:14:35.865218 371 consensus_peers.cc:485] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): Couldn't send request to peer 1f10f80f13c84e80b03eda7085a3b012. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b7b284cffcb04ce883a24f6a8796748f. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:35.865850 32640 raft_consensus.cc:1261] T b7b284cffcb04ce883a24f6a8796748f P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:35.867460 13959 consensus_queue.cc:1035] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:35.880626 14274 tablet_bootstrap.cc:492] T 88b57085de3144cea3f64922ae5ccf55 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:35.881110 14274 tablet_bootstrap.cc:492] T 88b57085de3144cea3f64922ae5ccf55 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap complete.
I20240503 02:14:35.881402 14274 ts_tablet_manager.cc:1367] T 88b57085de3144cea3f64922ae5ccf55 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.032s user 0.029s sys 0.000s
I20240503 02:14:35.882179 14274 raft_consensus.cc:348] T 88b57085de3144cea3f64922ae5ccf55 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:14:35.882400 14274 raft_consensus.cc:729] T 88b57085de3144cea3f64922ae5ccf55 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:14:35.882696 14274 consensus_queue.cc:260] T 88b57085de3144cea3f64922ae5ccf55 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:14:35.886364 14274 ts_tablet_manager.cc:1398] T 88b57085de3144cea3f64922ae5ccf55 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:14:35.887473 30156 tablet_copy_service.cc:342] P a9094240860e43bfb129193f3bfd5be6: Request end of tablet copy session fbbc2be6382e47908ecb7c68c367bd26-88b57085de3144cea3f64922ae5ccf55 received from {username='slave'} at 127.22.108.7:37671
I20240503 02:14:35.887616 30156 tablet_copy_service.cc:433] P a9094240860e43bfb129193f3bfd5be6: ending tablet copy session fbbc2be6382e47908ecb7c68c367bd26-88b57085de3144cea3f64922ae5ccf55 on tablet 88b57085de3144cea3f64922ae5ccf55 with peer fbbc2be6382e47908ecb7c68c367bd26
I20240503 02:14:35.901120 14291 raft_consensus.cc:2932] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:14:35.902407 32640 raft_consensus.cc:2932] T b7b284cffcb04ce883a24f6a8796748f P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:14:35.907148 306 raft_consensus.cc:2932] T b7b284cffcb04ce883a24f6a8796748f P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:14:35.909729 29733 catalog_manager.cc:5484] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: config changed from index -1 to 2, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:14:35.939648 306 raft_consensus.cc:1203] T 0007679ffe8047d79dd8ac778fb6f1ce P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 02:14:35.992292 32640 consensus_queue.cc:237] T 9cd1418eb38d420c9c41d660476eafb0 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
W20240503 02:14:35.998418 32574 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 9a4d92bc25374fb3960fde7d10be309d -> Peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Couldn't send request to peer 155735c7f8b14997ac4c770aee2a9401. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9cd1418eb38d420c9c41d660476eafb0. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:35.998603 565 raft_consensus.cc:1261] T 9cd1418eb38d420c9c41d660476eafb0 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:35.999173 14311 consensus_queue.cc:1035] T 9cd1418eb38d420c9c41d660476eafb0 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:35.999583 29872 raft_consensus.cc:1261] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer 9a4d92bc25374fb3960fde7d10be309d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:36.000126 14311 consensus_queue.cc:1035] T 9cd1418eb38d420c9c41d660476eafb0 P 9a4d92bc25374fb3960fde7d10be309d [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:36.012012 14174 raft_consensus.cc:2932] T 9cd1418eb38d420c9c41d660476eafb0 P 9a4d92bc25374fb3960fde7d10be309d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:14:36.012645 29872 raft_consensus.cc:2932] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:14:36.019923 29733 catalog_manager.cc:5484] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 reported cstate change: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New cstate: current_term: 1 leader_uuid: "9a4d92bc25374fb3960fde7d10be309d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:14:36.021152 565 raft_consensus.cc:2932] T 9cd1418eb38d420c9c41d660476eafb0 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:14:36.054267 483 maintenance_manager.cc:419] P fbbc2be6382e47908ecb7c68c367bd26: Scheduling UndoDeltaBlockGCOp(88b57085de3144cea3f64922ae5ccf55): 4582 bytes on disk
I20240503 02:14:36.056953 374 maintenance_manager.cc:643] P fbbc2be6382e47908ecb7c68c367bd26: UndoDeltaBlockGCOp(88b57085de3144cea3f64922ae5ccf55) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240503 02:14:36.069439 434 consensus_queue.cc:237] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:36.085410 305 raft_consensus.cc:1261] T 08e4d119d90f4146ba53a1b18efe2386 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:14:36.085592 29872 raft_consensus.cc:1261] T 08e4d119d90f4146ba53a1b18efe2386 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:14:36.085661 370 consensus_peers.cc:485] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Couldn't send request to peer a9094240860e43bfb129193f3bfd5be6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 08e4d119d90f4146ba53a1b18efe2386. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:36.086133 14270 consensus_queue.cc:1035] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:14:36.086452 13959 consensus_queue.cc:1035] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:14:36.094080 13959 raft_consensus.cc:2932] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:36.094663 29872 raft_consensus.cc:2932] T 08e4d119d90f4146ba53a1b18efe2386 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:36.098150 305 raft_consensus.cc:2932] T 08e4d119d90f4146ba53a1b18efe2386 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:36.101136 29727 catalog_manager.cc:5484] T 08e4d119d90f4146ba53a1b18efe2386 P 16440e365430405fb10366ab70e2b429 reported cstate change: config changed from index 3 to 4, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) added. New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:36.119411 565 raft_consensus.cc:1203] T 58e1aaecf47944569296fc1e0780efb5 P f575ce92c0224eeb80b9fae5f6561fbe [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.4->[2.5-2.5] Dedup: 2.5->[]
I20240503 02:14:36.146170 30004 consensus_queue.cc:237] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:14:36.148841 435 raft_consensus.cc:1261] T 35a5a7f4fe184b9fadc94c98b61deaf0 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:14:36.148943 29872 raft_consensus.cc:1261] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240503 02:14:36.149325 29940 consensus_peers.cc:485] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 35a5a7f4fe184b9fadc94c98b61deaf0. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:36.149626 14045 consensus_queue.cc:1035] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:14:36.150379 14046 consensus_queue.cc:1035] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:14:36.153770 14045 raft_consensus.cc:2932] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:36.157296 435 raft_consensus.cc:2932] T 35a5a7f4fe184b9fadc94c98b61deaf0 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:36.159111 29872 raft_consensus.cc:2932] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:36.163319 29729 catalog_manager.cc:5484] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: config changed from index 5 to 6, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:36.182417 14359 ts_tablet_manager.cc:902] T e80b43971dc246d9bfd5e1c603f8415c P 8a666e798fe8477d8dd1e51b385d5ead: Initiating tablet copy from peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:14:36.183836 14359 tablet_copy_client.cc:308] T e80b43971dc246d9bfd5e1c603f8415c P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.3:42679
I20240503 02:14:36.184510 30156 tablet_copy_service.cc:140] P a9094240860e43bfb129193f3bfd5be6: Received BeginTabletCopySession request for tablet e80b43971dc246d9bfd5e1c603f8415c from peer 8a666e798fe8477d8dd1e51b385d5ead ({username='slave'} at 127.22.108.6:41245)
I20240503 02:14:36.184698 30156 tablet_copy_service.cc:161] P a9094240860e43bfb129193f3bfd5be6: Beginning new tablet copy session on tablet e80b43971dc246d9bfd5e1c603f8415c from peer 8a666e798fe8477d8dd1e51b385d5ead at {username='slave'} at 127.22.108.6:41245: session id = 8a666e798fe8477d8dd1e51b385d5ead-e80b43971dc246d9bfd5e1c603f8415c
I20240503 02:14:36.187970 30156 tablet_copy_source_session.cc:205] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:14:36.191658 14359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e80b43971dc246d9bfd5e1c603f8415c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:36.201494 14359 tablet_copy_client.cc:783] T e80b43971dc246d9bfd5e1c603f8415c P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Starting download of 5 data blocks...
I20240503 02:14:36.209174 14359 tablet_copy_client.cc:647] T e80b43971dc246d9bfd5e1c603f8415c P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:36.211531 14359 tablet_copy_client.cc:523] T e80b43971dc246d9bfd5e1c603f8415c P 8a666e798fe8477d8dd1e51b385d5ead: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:36.212445 30135 raft_consensus.cc:1203] T 8929e26a53274cecbb808dbd8b203a64 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:14:36.215657 14359 tablet_bootstrap.cc:492] T e80b43971dc246d9bfd5e1c603f8415c P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap starting.
I20240503 02:14:36.260571 14359 tablet_bootstrap.cc:492] T e80b43971dc246d9bfd5e1c603f8415c P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:36.260963 14359 tablet_bootstrap.cc:492] T e80b43971dc246d9bfd5e1c603f8415c P 8a666e798fe8477d8dd1e51b385d5ead: Bootstrap complete.
I20240503 02:14:36.261248 14359 ts_tablet_manager.cc:1367] T e80b43971dc246d9bfd5e1c603f8415c P 8a666e798fe8477d8dd1e51b385d5ead: Time spent bootstrapping tablet: real 0.046s user 0.017s sys 0.004s
I20240503 02:14:36.262007 14359 raft_consensus.cc:348] T e80b43971dc246d9bfd5e1c603f8415c P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:36.262251 14359 raft_consensus.cc:729] T e80b43971dc246d9bfd5e1c603f8415c P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Initialized, Role: FOLLOWER
I20240503 02:14:36.262522 14359 consensus_queue.cc:260] T e80b43971dc246d9bfd5e1c603f8415c P 8a666e798fe8477d8dd1e51b385d5ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:36.263183 14359 ts_tablet_manager.cc:1398] T e80b43971dc246d9bfd5e1c603f8415c P 8a666e798fe8477d8dd1e51b385d5ead: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:14:36.264057 30156 tablet_copy_service.cc:342] P a9094240860e43bfb129193f3bfd5be6: Request end of tablet copy session 8a666e798fe8477d8dd1e51b385d5ead-e80b43971dc246d9bfd5e1c603f8415c received from {username='slave'} at 127.22.108.6:41245
I20240503 02:14:36.264165 30156 tablet_copy_service.cc:433] P a9094240860e43bfb129193f3bfd5be6: ending tablet copy session 8a666e798fe8477d8dd1e51b385d5ead-e80b43971dc246d9bfd5e1c603f8415c on tablet e80b43971dc246d9bfd5e1c603f8415c with peer 8a666e798fe8477d8dd1e51b385d5ead
I20240503 02:14:36.264667 14367 ts_tablet_manager.cc:902] T b90b887127f44c0d920225be10077c64 P 9a4d92bc25374fb3960fde7d10be309d: Initiating tablet copy from peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:14:36.266981 14367 tablet_copy_client.cc:308] T b90b887127f44c0d920225be10077c64 P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.4:33935
I20240503 02:14:36.267798 30286 tablet_copy_service.cc:140] P 1f10f80f13c84e80b03eda7085a3b012: Received BeginTabletCopySession request for tablet b90b887127f44c0d920225be10077c64 from peer 9a4d92bc25374fb3960fde7d10be309d ({username='slave'} at 127.22.108.5:44575)
I20240503 02:14:36.267966 30286 tablet_copy_service.cc:161] P 1f10f80f13c84e80b03eda7085a3b012: Beginning new tablet copy session on tablet b90b887127f44c0d920225be10077c64 from peer 9a4d92bc25374fb3960fde7d10be309d at {username='slave'} at 127.22.108.5:44575: session id = 9a4d92bc25374fb3960fde7d10be309d-b90b887127f44c0d920225be10077c64
I20240503 02:14:36.270748 30286 tablet_copy_source_session.cc:205] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:36.272205 14367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b90b887127f44c0d920225be10077c64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:36.278751 14367 tablet_copy_client.cc:783] T b90b887127f44c0d920225be10077c64 P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:36.279269 14367 tablet_copy_client.cc:647] T b90b887127f44c0d920225be10077c64 P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:36.280781 14367 tablet_copy_client.cc:523] T b90b887127f44c0d920225be10077c64 P 9a4d92bc25374fb3960fde7d10be309d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:36.284188 14367 tablet_bootstrap.cc:492] T b90b887127f44c0d920225be10077c64 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap starting.
I20240503 02:14:36.289899 14367 tablet_bootstrap.cc:492] T b90b887127f44c0d920225be10077c64 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:36.290251 14367 tablet_bootstrap.cc:492] T b90b887127f44c0d920225be10077c64 P 9a4d92bc25374fb3960fde7d10be309d: Bootstrap complete.
I20240503 02:14:36.290496 14367 ts_tablet_manager.cc:1367] T b90b887127f44c0d920225be10077c64 P 9a4d92bc25374fb3960fde7d10be309d: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20240503 02:14:36.291252 14367 raft_consensus.cc:348] T b90b887127f44c0d920225be10077c64 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:14:36.291478 14367 raft_consensus.cc:729] T b90b887127f44c0d920225be10077c64 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d92bc25374fb3960fde7d10be309d, State: Initialized, Role: FOLLOWER
I20240503 02:14:36.291740 14367 consensus_queue.cc:260] T b90b887127f44c0d920225be10077c64 P 9a4d92bc25374fb3960fde7d10be309d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:14:36.292315 14367 ts_tablet_manager.cc:1398] T b90b887127f44c0d920225be10077c64 P 9a4d92bc25374fb3960fde7d10be309d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:14:36.292943 30286 tablet_copy_service.cc:342] P 1f10f80f13c84e80b03eda7085a3b012: Request end of tablet copy session 9a4d92bc25374fb3960fde7d10be309d-b90b887127f44c0d920225be10077c64 received from {username='slave'} at 127.22.108.5:44575
I20240503 02:14:36.293157 30286 tablet_copy_service.cc:433] P 1f10f80f13c84e80b03eda7085a3b012: ending tablet copy session 9a4d92bc25374fb3960fde7d10be309d-b90b887127f44c0d920225be10077c64 on tablet b90b887127f44c0d920225be10077c64 with peer 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:14:36.305536 30135 consensus_queue.cc:237] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:14:36.307857 32640 raft_consensus.cc:1261] T b8b119c0d8044f2ab19b2100950270d6 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:36.308542 565 raft_consensus.cc:1261] T b8b119c0d8044f2ab19b2100950270d6 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:36.309144 14382 consensus_queue.cc:1035] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:36.309643 14382 consensus_queue.cc:1035] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:14:36.310024 30070 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): Couldn't send request to peer fbbc2be6382e47908ecb7c68c367bd26. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b8b119c0d8044f2ab19b2100950270d6. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:36.311729 353 maintenance_manager.cc:419] P 8a666e798fe8477d8dd1e51b385d5ead: Scheduling UndoDeltaBlockGCOp(e80b43971dc246d9bfd5e1c603f8415c): 4924 bytes on disk
I20240503 02:14:36.312500 32712 maintenance_manager.cc:643] P 8a666e798fe8477d8dd1e51b385d5ead: UndoDeltaBlockGCOp(e80b43971dc246d9bfd5e1c603f8415c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":122,"lbm_reads_lt_1ms":4}
I20240503 02:14:36.328600 13556 raft_consensus.cc:2932] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:14:36.329000 565 raft_consensus.cc:2932] T b8b119c0d8044f2ab19b2100950270d6 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:14:36.333815 32640 raft_consensus.cc:2932] T b8b119c0d8044f2ab19b2100950270d6 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:14:36.335883 29729 catalog_manager.cc:5484] T b8b119c0d8044f2ab19b2100950270d6 P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: config changed from index -1 to 2, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:14:36.377490 30004 consensus_queue.cc:237] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:14:36.379954 305 raft_consensus.cc:1261] T 6635aacafed54994965ad48b2df4f163 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:14:36.380339 30265 raft_consensus.cc:1261] T 6635aacafed54994965ad48b2df4f163 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:14:36.380825 14046 consensus_queue.cc:1035] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 02:14:36.381497 29940 consensus_peers.cc:485] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6635aacafed54994965ad48b2df4f163. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:36.381525 14045 consensus_queue.cc:1035] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:14:36.383932 14046 raft_consensus.cc:2932] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:36.384536 30265 raft_consensus.cc:2932] T 6635aacafed54994965ad48b2df4f163 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:36.388626 306 raft_consensus.cc:2932] T 6635aacafed54994965ad48b2df4f163 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:36.392531 29727 catalog_manager.cc:5484] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: config changed from index 5 to 6, VOTER 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) added. New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:36.421048 435 raft_consensus.cc:1203] T 88b57085de3144cea3f64922ae5ccf55 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 02:14:36.432502 30312 maintenance_manager.cc:419] P 1f10f80f13c84e80b03eda7085a3b012: Scheduling FlushMRSOp(fa288463ae154e28afd9c71b4b55ef3a): perf score=0.033340
I20240503 02:14:36.474871 29921 maintenance_manager.cc:419] P 16440e365430405fb10366ab70e2b429: Scheduling FlushMRSOp(df37d88053e2419fbe40d03023d0c756): perf score=0.033350
I20240503 02:14:36.514730 30203 maintenance_manager.cc:643] P 1f10f80f13c84e80b03eda7085a3b012: FlushMRSOp(fa288463ae154e28afd9c71b4b55ef3a) complete. Timing: real 0.082s user 0.057s sys 0.022s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":17072,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":1819,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":821,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":23378,"threads_started":2,"wal-append.queue_time_us":6290}
I20240503 02:14:36.517979 30312 maintenance_manager.cc:419] P 1f10f80f13c84e80b03eda7085a3b012: Scheduling FlushMRSOp(df37d88053e2419fbe40d03023d0c756): perf score=0.033359
I20240503 02:14:36.531970 14414 ts_tablet_manager.cc:902] T b7b284cffcb04ce883a24f6a8796748f P 1f10f80f13c84e80b03eda7085a3b012: Initiating tablet copy from peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:14:36.556090 30052 maintenance_manager.cc:419] P 155735c7f8b14997ac4c770aee2a9401: Scheduling FlushMRSOp(7076453cbe5944eb9b075127e1e0495d): perf score=0.033376
I20240503 02:14:36.585973 30203 maintenance_manager.cc:643] P 1f10f80f13c84e80b03eda7085a3b012: FlushMRSOp(df37d88053e2419fbe40d03023d0c756) complete. Timing: real 0.068s user 0.057s sys 0.002s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":1922,"drs_written":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1010,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":10829,"threads_started":1,"wal-append.queue_time_us":10701}
I20240503 02:14:36.589107 30312 maintenance_manager.cc:419] P 1f10f80f13c84e80b03eda7085a3b012: Scheduling UndoDeltaBlockGCOp(df37d88053e2419fbe40d03023d0c756): 4923 bytes on disk
I20240503 02:14:36.590492 30203 maintenance_manager.cc:643] P 1f10f80f13c84e80b03eda7085a3b012: UndoDeltaBlockGCOp(df37d88053e2419fbe40d03023d0c756) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240503 02:14:36.593469 30312 maintenance_manager.cc:419] P 1f10f80f13c84e80b03eda7085a3b012: Scheduling UndoDeltaBlockGCOp(fa288463ae154e28afd9c71b4b55ef3a): 4583 bytes on disk
I20240503 02:14:36.594408 14425 ts_tablet_manager.cc:902] T 9cd1418eb38d420c9c41d660476eafb0 P 155735c7f8b14997ac4c770aee2a9401: Initiating tablet copy from peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:14:36.595543 30203 maintenance_manager.cc:643] P 1f10f80f13c84e80b03eda7085a3b012: UndoDeltaBlockGCOp(fa288463ae154e28afd9c71b4b55ef3a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20240503 02:14:36.608824 14414 tablet_copy_client.cc:308] T b7b284cffcb04ce883a24f6a8796748f P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.7:44241
I20240503 02:14:36.609706 457 tablet_copy_service.cc:140] P fbbc2be6382e47908ecb7c68c367bd26: Received BeginTabletCopySession request for tablet b7b284cffcb04ce883a24f6a8796748f from peer 1f10f80f13c84e80b03eda7085a3b012 ({username='slave'} at 127.22.108.4:42187)
I20240503 02:14:36.609907 457 tablet_copy_service.cc:161] P fbbc2be6382e47908ecb7c68c367bd26: Beginning new tablet copy session on tablet b7b284cffcb04ce883a24f6a8796748f from peer 1f10f80f13c84e80b03eda7085a3b012 at {username='slave'} at 127.22.108.4:42187: session id = 1f10f80f13c84e80b03eda7085a3b012-b7b284cffcb04ce883a24f6a8796748f
I20240503 02:14:36.613481 457 tablet_copy_source_session.cc:205] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:36.614910 14414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7b284cffcb04ce883a24f6a8796748f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:36.622859 14414 tablet_copy_client.cc:783] T b7b284cffcb04ce883a24f6a8796748f P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:36.623198 14414 tablet_copy_client.cc:647] T b7b284cffcb04ce883a24f6a8796748f P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:36.627054 14414 tablet_copy_client.cc:523] T b7b284cffcb04ce883a24f6a8796748f P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:36.630702 14430 ts_tablet_manager.cc:902] T 08e4d119d90f4146ba53a1b18efe2386 P a9094240860e43bfb129193f3bfd5be6: Initiating tablet copy from peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:14:36.631605 14414 tablet_bootstrap.cc:492] T b7b284cffcb04ce883a24f6a8796748f P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
I20240503 02:14:36.652809 14414 tablet_bootstrap.cc:492] T b7b284cffcb04ce883a24f6a8796748f P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:36.655308 14414 tablet_bootstrap.cc:492] T b7b284cffcb04ce883a24f6a8796748f P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap complete.
I20240503 02:14:36.656430 14414 ts_tablet_manager.cc:1367] T b7b284cffcb04ce883a24f6a8796748f P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.005s
I20240503 02:14:36.657784 14414 raft_consensus.cc:348] T b7b284cffcb04ce883a24f6a8796748f P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:14:36.658429 14425 tablet_copy_client.cc:308] T 9cd1418eb38d420c9c41d660476eafb0 P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.5:35595
I20240503 02:14:36.658757 14414 raft_consensus.cc:729] T b7b284cffcb04ce883a24f6a8796748f P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:14:36.659089 14414 consensus_queue.cc:260] T b7b284cffcb04ce883a24f6a8796748f P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:14:36.659261 32662 tablet_copy_service.cc:140] P 9a4d92bc25374fb3960fde7d10be309d: Received BeginTabletCopySession request for tablet 9cd1418eb38d420c9c41d660476eafb0 from peer 155735c7f8b14997ac4c770aee2a9401 ({username='slave'} at 127.22.108.2:49531)
I20240503 02:14:36.659446 32662 tablet_copy_service.cc:161] P 9a4d92bc25374fb3960fde7d10be309d: Beginning new tablet copy session on tablet 9cd1418eb38d420c9c41d660476eafb0 from peer 155735c7f8b14997ac4c770aee2a9401 at {username='slave'} at 127.22.108.2:49531: session id = 155735c7f8b14997ac4c770aee2a9401-9cd1418eb38d420c9c41d660476eafb0
I20240503 02:14:36.659814 14414 ts_tablet_manager.cc:1398] T b7b284cffcb04ce883a24f6a8796748f P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 02:14:36.662218 32662 tablet_copy_source_session.cc:205] T 9cd1418eb38d420c9c41d660476eafb0 P 9a4d92bc25374fb3960fde7d10be309d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:36.662741 457 tablet_copy_service.cc:342] P fbbc2be6382e47908ecb7c68c367bd26: Request end of tablet copy session 1f10f80f13c84e80b03eda7085a3b012-b7b284cffcb04ce883a24f6a8796748f received from {username='slave'} at 127.22.108.4:42187
I20240503 02:14:36.662866 457 tablet_copy_service.cc:433] P fbbc2be6382e47908ecb7c68c367bd26: ending tablet copy session 1f10f80f13c84e80b03eda7085a3b012-b7b284cffcb04ce883a24f6a8796748f on tablet b7b284cffcb04ce883a24f6a8796748f with peer 1f10f80f13c84e80b03eda7085a3b012
I20240503 02:14:36.663965 14425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cd1418eb38d420c9c41d660476eafb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:36.671614 14425 tablet_copy_client.cc:783] T 9cd1418eb38d420c9c41d660476eafb0 P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:36.671975 14425 tablet_copy_client.cc:647] T 9cd1418eb38d420c9c41d660476eafb0 P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:36.672277 14430 tablet_copy_client.cc:308] T 08e4d119d90f4146ba53a1b18efe2386 P a9094240860e43bfb129193f3bfd5be6: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.7:44241
I20240503 02:14:36.674085 457 tablet_copy_service.cc:140] P fbbc2be6382e47908ecb7c68c367bd26: Received BeginTabletCopySession request for tablet 08e4d119d90f4146ba53a1b18efe2386 from peer a9094240860e43bfb129193f3bfd5be6 ({username='slave'} at 127.22.108.3:42051)
I20240503 02:14:36.674261 457 tablet_copy_service.cc:161] P fbbc2be6382e47908ecb7c68c367bd26: Beginning new tablet copy session on tablet 08e4d119d90f4146ba53a1b18efe2386 from peer a9094240860e43bfb129193f3bfd5be6 at {username='slave'} at 127.22.108.3:42051: session id = a9094240860e43bfb129193f3bfd5be6-08e4d119d90f4146ba53a1b18efe2386
I20240503 02:14:36.677371 457 tablet_copy_source_session.cc:205] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:36.677919 14425 tablet_copy_client.cc:523] T 9cd1418eb38d420c9c41d660476eafb0 P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:36.678874 14430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08e4d119d90f4146ba53a1b18efe2386. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:36.693372 14367 ts_tablet_manager.cc:902] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 9a4d92bc25374fb3960fde7d10be309d: Initiating tablet copy from peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
F20240503 02:14:36.693800 14367 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0007679ffe8047d79dd8ac778fb6f1ce.queue_time_us, tablet-download-pool-0007679ffe8047d79dd8ac778fb6f1ce.queue_time_us) (tablet-download-pool-0007679ffe8047d79dd8ac778fb6f1ce.run_cpu_time_us, tablet-download-pool-0007679ffe8047d79dd8ac778fb6f1ce.run_cpu_time_us) (tablet-download-pool-0007679ffe8047d79dd8ac778fb6f1ce.run_wall_time_us, tablet-download-pool-0007679ffe8047d79dd8ac778fb6f1ce.run_wall_time_us) (tablet-download-pool-04e47da8cf7445dcacaf1f7c8950bd34.queue_time_us, tablet-download-pool-04e47da8cf7445dcacaf1f7c8950bd34.queue_time_us) (tablet-download-pool-04e47da8cf7445dcacaf1f7c8950bd34.run_cpu_time_us, tablet-download-pool-04e47da8cf7445dcacaf1f7c8950bd34.run_cpu_time_us) (tablet-download-pool-04e47da8cf7445dcacaf1f7c8950bd34.run_wall_time_us, tablet-download-pool-04e47da8cf7445dcacaf1f7c8950bd34.run_wall_time_us) (tablet-download-pool-35a5a7f4fe184b9fadc94c98b61deaf0.queue_time_us, tablet-download-pool-35a5a7f4fe184b9fadc94c98b61deaf0.queue_time_us) (tablet-download-pool-3f00949b4ca84a49b0f6e62bd7373586.queue_time_us, tablet-download-pool-3f00949b4ca84a49b0f6e62bd7373586.queue_time_us) (tablet-download-pool-3f00949b4ca84a49b0f6e62bd7373586.run_cpu_time_us, tablet-download-pool-3f00949b4ca84a49b0f6e62bd7373586.run_cpu_time_us) (tablet-download-pool-3f00949b4ca84a49b0f6e62bd7373586.run_wall_time_us, tablet-download-pool-3f00949b4ca84a49b0f6e62bd7373586.run_wall_time_us) (tablet-download-pool-480f7b57c2c04c1399b945a023fbe41f.queue_time_us, tablet-download-pool-480f7b57c2c04c1399b945a023fbe41f.queue_time_us) (tablet-download-pool-480f7b57c2c04c1399b945a023fbe41f.run_cpu_time_us, tablet-download-pool-480f7b57c2c04c1399b945a023fbe41f.run_cpu_time_us) (tablet-download-pool-480f7b57c2c04c1399b945a023fbe41f.run_wall_time_us, tablet-download-pool-480f7b57c2c04c1399b945a023fbe41f.run_wall_time_us) (tablet-download-pool-493c813acc27492fb952791cf2dc12a2.queue_time_us, tablet-download-pool-493c813acc27492fb952791cf2dc12a2.queue_time_us) (tablet-download-pool-493c813acc27492fb952791cf2dc12a2.run_cpu_time_us, tablet-download-pool-493c813acc27492fb952791cf2dc12a2.run_cpu_time_us) (tablet-download-pool-493c813acc27492fb952791cf2dc12a2.run_wall_time_us, tablet-download-pool-493c813acc27492fb952791cf2dc12a2.run_wall_time_us) (tablet-download-pool-6bd4fcc4faf44d148a03beaafc1c05ba.queue_time_us, tablet-download-pool-6bd4fcc4faf44d148a03beaafc1c05ba.queue_time_us) (tablet-download-pool-6bd4fcc4faf44d148a03beaafc1c05ba.run_cpu_time_us, tablet-download-pool-6bd4fcc4faf44d148a03beaafc1c05ba.run_cpu_time_us) (tablet-download-pool-6bd4fcc4faf44d148a03beaafc1c05ba.run_wall_time_us, tablet-download-pool-6bd4fcc4faf44d148a03beaafc1c05ba.run_wall_time_us) (tablet-download-pool-6bf84757d1fb44689da830a82e857787.queue_time_us, tablet-download-pool-6bf84757d1fb44689da830a82e857787.queue_time_us) (tablet-download-pool-6bf84757d1fb44689da830a82e857787.run_cpu_time_us, tablet-download-pool-6bf84757d1fb44689da830a82e857787.run_cpu_time_us) (tablet-download-pool-6bf84757d1fb44689da830a82e857787.run_wall_time_us, tablet-download-pool-6bf84757d1fb44689da830a82e857787.run_wall_time_us) (tablet-download-pool-7076453cbe5944eb9b075127e1e0495d.queue_time_us, tablet-download-pool-7076453cbe5944eb9b075127e1e0495d.queue_time_us) (tablet-download-pool-7076453cbe5944eb9b075127e1e0495d.run_cpu_time_us, tablet-download-pool-7076453cbe5944eb9b075127e1e0495d.run_cpu_time_us) (tablet-download-pool-7076453cbe5944eb9b075127e1e0495d.run_wall_time_us, tablet-download-pool-7076453cbe5944eb9b075127e1e0495d.run_wall_time_us) (tablet-download-pool-8f5e6ff35b994d46a1e1fe216eb356ce.queue_time_us, tablet-download-pool-8f5e6ff35b994d46a1e1fe216eb356ce.queue_time_us) (tablet-download-pool-8f5e6ff35b994d46a1e1fe216eb356ce.run_cpu_time_us, tablet-download-pool-8f5e6ff35b994d46a1e1fe216eb356ce.run_cpu_time_us) (tablet-download-pool-8f5e6ff35b994d46a1e1fe216eb356ce.run_wall_time_us, tablet-download-pool-8f5e6ff35b994d46a1e1fe216eb356ce.run_wall_time_us) (tablet-download-pool-8fdbaf1df10b46d5a47326415d0cd0e9.queue_time_us, tablet-download-pool-8fdbaf1df10b46d5a47326415d0cd0e9.queue_time_us) (tablet-download-pool-8fdbaf1df10b46d5a47326415d0cd0e9.run_cpu_time_us, tablet-download-pool-8fdbaf1df10b46d5a47326415d0cd0e9.run_cpu_time_us) (tablet-download-pool-8fdbaf1df10b46d5a47326415d0cd0e9.run_wall_time_us, tablet-download-pool-8fdbaf1df10b46d5a47326415d0cd0e9.run_wall_time_us) (tablet-download-pool-b90b887127f44c0d920225be10077c64.queue_time_us, tablet-download-pool-b90b887127f44c0d920225be10077c64.queue_time_us) (tablet-download-pool-b90b887127f44c0d920225be10077c64.run_cpu_time_us, tablet-download-pool-b90b887127f44c0d920225be10077c64.run_cpu_time_us) (tablet-download-pool-b90b887127f44c0d920225be10077c64.run_wall_time_us, tablet-download-pool-b90b887127f44c0d920225be10077c64.run_wall_time_us) (tablet-download-pool-ba97049157a646c780fa203d34cd4c82.queue_time_us, tablet-download-pool-ba97049157a646c780fa203d34cd4c82.queue_time_us) (tablet-download-pool-ba97049157a646c780fa203d34cd4c82.run_cpu_time_us, tablet-download-pool-ba97049157a646c780fa203d34cd4c82.run_cpu_time_us) (tablet-download-pool-ba97049157a646c780fa203d34cd4c82.run_wall_time_us, tablet-download-pool-ba97049157a646c780fa203d34cd4c82.run_wall_time_us) (tablet-download-pool-c5cd64614c53487ea1c6cc4a4ca5c379.queue_time_us, tablet-download-pool-c5cd64614c53487ea1c6cc4a4ca5c379.queue_time_us) (tablet-download-pool-c5cd64614c53487ea1c6cc4a4ca5c379.run_cpu_time_us, tablet-download-pool-c5cd64614c53487ea1c6cc4a4ca5c379.run_cpu_time_us) (tablet-download-pool-c5cd64614c53487ea1c6cc4a4ca5c379.run_wall_time_us, tablet-download-pool-c5cd64614c53487ea1c6cc4a4ca5c379.run_wall_time_us) (tablet-download-pool-e80b43971dc246d9bfd5e1c603f8415c.queue_time_us, tablet-download-pool-e80b43971dc246d9bfd5e1c603f8415c.queue_time_us) (tablet-download-pool-e80b43971dc246d9bfd5e1c603f8415c.run_cpu_time_us, tablet-download-pool-e80b43971dc246d9bfd5e1c603f8415c.run_cpu_time_us) (tablet-download-pool-e80b43971dc246d9bfd5e1c603f8415c.run_wall_time_us, tablet-download-pool-e80b43971dc246d9bfd5e1c603f8415c.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240503 02:14:36.703040 14425 tablet_bootstrap.cc:492] T 9cd1418eb38d420c9c41d660476eafb0 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
@ 0x7f594cbb4dcd google::LogMessage::Fail() at ??:0
I20240503 02:14:36.707033 14430 tablet_copy_client.cc:783] T 08e4d119d90f4146ba53a1b18efe2386 P a9094240860e43bfb129193f3bfd5be6: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:36.707285 14430 tablet_copy_client.cc:647] T 08e4d119d90f4146ba53a1b18efe2386 P a9094240860e43bfb129193f3bfd5be6: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:36.708330 30182 maintenance_manager.cc:419] P a9094240860e43bfb129193f3bfd5be6: Scheduling FlushMRSOp(7076453cbe5944eb9b075127e1e0495d): perf score=0.033396
I20240503 02:14:36.710889 14430 tablet_copy_client.cc:523] T 08e4d119d90f4146ba53a1b18efe2386 P a9094240860e43bfb129193f3bfd5be6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7f594cbb8b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f594cbb47cc google::LogMessage::Flush() at ??:0
I20240503 02:14:36.719429 14430 tablet_bootstrap.cc:492] T 08e4d119d90f4146ba53a1b18efe2386 P a9094240860e43bfb129193f3bfd5be6: Bootstrap starting.
@ 0x7f594cbb5f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240503 02:14:36.734669 306 raft_consensus.cc:1203] T e80b43971dc246d9bfd5e1c603f8415c P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 02:14:36.741452 14425 tablet_bootstrap.cc:492] T 9cd1418eb38d420c9c41d660476eafb0 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:36.741991 14425 tablet_bootstrap.cc:492] T 9cd1418eb38d420c9c41d660476eafb0 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap complete.
I20240503 02:14:36.742252 14425 ts_tablet_manager.cc:1367] T 9cd1418eb38d420c9c41d660476eafb0 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.039s user 0.006s sys 0.005s
I20240503 02:14:36.746472 14425 raft_consensus.cc:348] T 9cd1418eb38d420c9c41d660476eafb0 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:14:36.746758 14425 raft_consensus.cc:729] T 9cd1418eb38d420c9c41d660476eafb0 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:14:36.751513 14425 consensus_queue.cc:260] T 9cd1418eb38d420c9c41d660476eafb0 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:14:36.755131 14425 ts_tablet_manager.cc:1398] T 9cd1418eb38d420c9c41d660476eafb0 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 02:14:36.756548 14430 tablet_bootstrap.cc:492] T 08e4d119d90f4146ba53a1b18efe2386 P a9094240860e43bfb129193f3bfd5be6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:36.756621 32662 tablet_copy_service.cc:342] P 9a4d92bc25374fb3960fde7d10be309d: Request end of tablet copy session 155735c7f8b14997ac4c770aee2a9401-9cd1418eb38d420c9c41d660476eafb0 received from {username='slave'} at 127.22.108.2:49531
I20240503 02:14:36.756769 32662 tablet_copy_service.cc:433] P 9a4d92bc25374fb3960fde7d10be309d: ending tablet copy session 155735c7f8b14997ac4c770aee2a9401-9cd1418eb38d420c9c41d660476eafb0 on tablet 9cd1418eb38d420c9c41d660476eafb0 with peer 155735c7f8b14997ac4c770aee2a9401
I20240503 02:14:36.757047 14430 tablet_bootstrap.cc:492] T 08e4d119d90f4146ba53a1b18efe2386 P a9094240860e43bfb129193f3bfd5be6: Bootstrap complete.
I20240503 02:14:36.761765 14430 ts_tablet_manager.cc:1367] T 08e4d119d90f4146ba53a1b18efe2386 P a9094240860e43bfb129193f3bfd5be6: Time spent bootstrapping tablet: real 0.043s user 0.006s sys 0.005s
I20240503 02:14:36.762542 14430 raft_consensus.cc:348] T 08e4d119d90f4146ba53a1b18efe2386 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:36.762787 14430 raft_consensus.cc:729] T 08e4d119d90f4146ba53a1b18efe2386 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Initialized, Role: FOLLOWER
I20240503 02:14:36.763031 14430 consensus_queue.cc:260] T 08e4d119d90f4146ba53a1b18efe2386 P a9094240860e43bfb129193f3bfd5be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:36.765271 14430 ts_tablet_manager.cc:1398] T 08e4d119d90f4146ba53a1b18efe2386 P a9094240860e43bfb129193f3bfd5be6: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:14:36.766057 457 tablet_copy_service.cc:342] P fbbc2be6382e47908ecb7c68c367bd26: Request end of tablet copy session a9094240860e43bfb129193f3bfd5be6-08e4d119d90f4146ba53a1b18efe2386 received from {username='slave'} at 127.22.108.3:42051
I20240503 02:14:36.766181 457 tablet_copy_service.cc:433] P fbbc2be6382e47908ecb7c68c367bd26: ending tablet copy session a9094240860e43bfb129193f3bfd5be6-08e4d119d90f4146ba53a1b18efe2386 on tablet 08e4d119d90f4146ba53a1b18efe2386 with peer a9094240860e43bfb129193f3bfd5be6
@ 0x7f594f8f5742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f594f8c1f8b kudu::ThreadPool::ThreadPool() at ??:0
I20240503 02:14:36.810954 29812 maintenance_manager.cc:643] P 16440e365430405fb10366ab70e2b429: FlushMRSOp(df37d88053e2419fbe40d03023d0c756) complete. Timing: real 0.336s user 0.047s sys 0.016s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":312,"dirs.run_cpu_time_us":245,"dirs.run_wall_time_us":2603,"drs_written":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4507,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":29686,"threads_started":2,"wal-append.queue_time_us":8167}
@ 0x7f594f8bbcd6 kudu::ThreadPoolBuilder::Build() at ??:0
I20240503 02:14:36.814800 29921 maintenance_manager.cc:419] P 16440e365430405fb10366ab70e2b429: Scheduling FlushMRSOp(fa288463ae154e28afd9c71b4b55ef3a): perf score=0.033442
I20240503 02:14:36.825155 32640 raft_consensus.cc:1203] T b90b887127f44c0d920225be10077c64 P 9a4d92bc25374fb3960fde7d10be309d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
@ 0x7f595ed0f8de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240503 02:14:36.833071 30073 maintenance_manager.cc:643] P a9094240860e43bfb129193f3bfd5be6: FlushMRSOp(7076453cbe5944eb9b075127e1e0495d) complete. Timing: real 0.124s user 0.048s sys 0.003s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":1661,"dirs.run_cpu_time_us":358,"dirs.run_wall_time_us":1640,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5661,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":16055,"threads_started":2,"wal-append.queue_time_us":8203}
I20240503 02:14:36.836937 30182 maintenance_manager.cc:419] P a9094240860e43bfb129193f3bfd5be6: Scheduling UndoDeltaBlockGCOp(7076453cbe5944eb9b075127e1e0495d): 4838 bytes on disk
I20240503 02:14:36.844295 30073 maintenance_manager.cc:643] P a9094240860e43bfb129193f3bfd5be6: UndoDeltaBlockGCOp(7076453cbe5944eb9b075127e1e0495d) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":1938,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3}
@ 0x7f595ed27daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f595ee52480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240503 02:14:36.864534 14457 ts_tablet_manager.cc:902] T 6635aacafed54994965ad48b2df4f163 P 9a4d92bc25374fb3960fde7d10be309d: Initiating tablet copy from peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:14:36.865816 14458 ts_tablet_manager.cc:902] T b8b119c0d8044f2ab19b2100950270d6 P fbbc2be6382e47908ecb7c68c367bd26: Initiating tablet copy from peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:14:36.874874 29943 maintenance_manager.cc:643] P 155735c7f8b14997ac4c770aee2a9401: FlushMRSOp(7076453cbe5944eb9b075127e1e0495d) complete. Timing: real 0.318s user 0.059s sys 0.010s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":8049,"dirs.run_cpu_time_us":217,"dirs.run_wall_time_us":3107,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":967,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":35530,"threads_started":2,"wal-append.queue_time_us":22257}
I20240503 02:14:36.878888 30052 maintenance_manager.cc:419] P 155735c7f8b14997ac4c770aee2a9401: Scheduling UndoDeltaBlockGCOp(7076453cbe5944eb9b075127e1e0495d): 4838 bytes on disk
I20240503 02:14:36.880127 29943 maintenance_manager.cc:643] P 155735c7f8b14997ac4c770aee2a9401: UndoDeltaBlockGCOp(7076453cbe5944eb9b075127e1e0495d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":139,"lbm_reads_lt_1ms":4}
@ 0x7f595ee8aa8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f595ee6d981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
I20240503 02:14:36.891391 14458 tablet_copy_client.cc:308] T b8b119c0d8044f2ab19b2100950270d6 P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.3:42679
I20240503 02:14:36.892175 30156 tablet_copy_service.cc:140] P a9094240860e43bfb129193f3bfd5be6: Received BeginTabletCopySession request for tablet b8b119c0d8044f2ab19b2100950270d6 from peer fbbc2be6382e47908ecb7c68c367bd26 ({username='slave'} at 127.22.108.7:37671)
I20240503 02:14:36.892361 30156 tablet_copy_service.cc:161] P a9094240860e43bfb129193f3bfd5be6: Beginning new tablet copy session on tablet b8b119c0d8044f2ab19b2100950270d6 from peer fbbc2be6382e47908ecb7c68c367bd26 at {username='slave'} at 127.22.108.7:37671: session id = fbbc2be6382e47908ecb7c68c367bd26-b8b119c0d8044f2ab19b2100950270d6
I20240503 02:14:36.896888 30156 tablet_copy_source_session.cc:205] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:36.901999 14458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8b119c0d8044f2ab19b2100950270d6. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f595ee6d70e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240503 02:14:36.909722 14458 tablet_copy_client.cc:783] T b8b119c0d8044f2ab19b2100950270d6 P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:36.909984 14458 tablet_copy_client.cc:647] T b8b119c0d8044f2ab19b2100950270d6 P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:36.911834 14458 tablet_copy_client.cc:523] T b8b119c0d8044f2ab19b2100950270d6 P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:36.918213 14458 tablet_bootstrap.cc:492] T b8b119c0d8044f2ab19b2100950270d6 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
@ 0x7f594f8c5b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f594f8c919c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f594f8c9019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f594f8a401e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f5952c296db start_thread at ??:0
@ 0x7f594ba6771f clone at ??:0
I20240503 02:14:36.952242 14458 tablet_bootstrap.cc:492] T b8b119c0d8044f2ab19b2100950270d6 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:36.952593 14458 tablet_bootstrap.cc:492] T b8b119c0d8044f2ab19b2100950270d6 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap complete.
I20240503 02:14:36.952788 14458 ts_tablet_manager.cc:1367] T b8b119c0d8044f2ab19b2100950270d6 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.004s
I20240503 02:14:36.953666 14458 raft_consensus.cc:348] T b8b119c0d8044f2ab19b2100950270d6 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:14:36.953907 14458 raft_consensus.cc:729] T b8b119c0d8044f2ab19b2100950270d6 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:14:36.954237 14458 consensus_queue.cc:260] T b8b119c0d8044f2ab19b2100950270d6 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:14:36.955098 14458 ts_tablet_manager.cc:1398] T b8b119c0d8044f2ab19b2100950270d6 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:14:36.956033 30156 tablet_copy_service.cc:342] P a9094240860e43bfb129193f3bfd5be6: Request end of tablet copy session fbbc2be6382e47908ecb7c68c367bd26-b8b119c0d8044f2ab19b2100950270d6 received from {username='slave'} at 127.22.108.7:37671
I20240503 02:14:36.956168 30156 tablet_copy_service.cc:433] P a9094240860e43bfb129193f3bfd5be6: ending tablet copy session fbbc2be6382e47908ecb7c68c367bd26-b8b119c0d8044f2ab19b2100950270d6 on tablet b8b119c0d8044f2ab19b2100950270d6 with peer fbbc2be6382e47908ecb7c68c367bd26
I20240503 02:14:36.987569 29812 maintenance_manager.cc:643] P 16440e365430405fb10366ab70e2b429: FlushMRSOp(fa288463ae154e28afd9c71b4b55ef3a) complete. Timing: real 0.173s user 0.057s sys 0.019s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":47,"dirs.run_cpu_time_us":226,"dirs.run_wall_time_us":3355,"drs_written":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4,"lbm_write_time_us":899,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":6437,"threads_started":1,"wal-append.queue_time_us":6324}
I20240503 02:14:36.991477 29921 maintenance_manager.cc:419] P 16440e365430405fb10366ab70e2b429: Scheduling UndoDeltaBlockGCOp(df37d88053e2419fbe40d03023d0c756): 4923 bytes on disk
I20240503 02:14:36.992082 29812 maintenance_manager.cc:643] P 16440e365430405fb10366ab70e2b429: UndoDeltaBlockGCOp(df37d88053e2419fbe40d03023d0c756) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20240503 02:14:37.003158 29921 maintenance_manager.cc:419] P 16440e365430405fb10366ab70e2b429: Scheduling UndoDeltaBlockGCOp(fa288463ae154e28afd9c71b4b55ef3a): 4583 bytes on disk
I20240503 02:14:37.003767 29812 maintenance_manager.cc:643] P 16440e365430405fb10366ab70e2b429: UndoDeltaBlockGCOp(fa288463ae154e28afd9c71b4b55ef3a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240503 02:14:37.052213 30266 raft_consensus.cc:1203] T b7b284cffcb04ce883a24f6a8796748f P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:14:37.142453 30134 raft_consensus.cc:1203] T 08e4d119d90f4146ba53a1b18efe2386 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:14:37.256362 435 raft_consensus.cc:1203] T b8b119c0d8044f2ab19b2100950270d6 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:14:37.371093 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:37.409989 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:37.433542 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:37.446334 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:37.460366 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:37.495934 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:37.503938 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240503 02:14:37.523175 29809 connection.cc:535] client connection to 127.22.108.5:35595 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:14:37.523447 29809 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:14:37.529698 30070 connection.cc:535] client connection to 127.22.108.5:35595 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:14:37.529773 32709 connection.cc:535] client connection to 127.22.108.5:35595 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:14:37.529990 32709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:14:37.529950 30070 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:14:37.531921 29940 connection.cc:535] client connection to 127.22.108.5:35595 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:14:37.532198 29940 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:14:37.549865 30200 connection.cc:535] client connection to 127.22.108.5:35595 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:14:37.550143 30200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:14:37.550628 502 connection.cc:535] client connection to 127.22.108.5:35595 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:14:37.550839 502 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:14:37.551263 371 connection.cc:535] client connection to 127.22.108.5:35595 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:14:37.551434 371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:14:37.569588 30070 consensus_peers.cc:485] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:14:37.569772 30070 consensus_peers.cc:485] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:14:37.569852 30070 consensus_peers.cc:485] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:14:37.569916 30070 consensus_peers.cc:485] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:14:37.569981 30070 consensus_peers.cc:485] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:14:37.570053 30070 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:14:37.572796 502 consensus_peers.cc:485] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:14:37.572974 502 consensus_peers.cc:485] T ae4373a66af14c8f8a633963dffd2261 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:14:37.573114 502 consensus_peers.cc:485] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:14:37.573300 502 consensus_peers.cc:485] T 36d3f1edbab7477f9b7ac400160491a0 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:14:37.573510 502 consensus_peers.cc:485] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:14:37.574282 371 consensus_peers.cc:485] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:14:37.574440 371 consensus_peers.cc:485] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:14:37.574601 371 consensus_peers.cc:485] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:14:37.584362 29809 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.5:35595, user_credentials={real_user=slave}} blocked reactor thread for 60941.9us
W20240503 02:14:37.591040 32709 consensus_peers.cc:485] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:14:37.591207 32709 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:14:37.595686 30200 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.5:35595, user_credentials={real_user=slave}} blocked reactor thread for 45598us
W20240503 02:14:37.597410 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:37.597517 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:37.597581 29940 consensus_peers.cc:485] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:14:37.597666 29940 consensus_peers.cc:485] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:14:37.608170 29809 consensus_peers.cc:485] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:14:37.608582 29809 consensus_peers.cc:485] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:14:37.608940 29809 consensus_peers.cc:485] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:14:37.611256 29809 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:14:37.613020 30200 consensus_peers.cc:485] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:14:37.613178 30200 consensus_peers.cc:485] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:14:37.613304 30200 consensus_peers.cc:485] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:14:37.613404 30200 consensus_peers.cc:485] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:14:37.683465 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:37.854929 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:37.866200 14545 raft_consensus.cc:482] T 0ec95509de3443bd96bf91a08ef7df4a P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:37.866384 14545 raft_consensus.cc:504] T 0ec95509de3443bd96bf91a08ef7df4a P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:14:37.867161 14545 leader_election.cc:288] T 0ec95509de3443bd96bf91a08ef7df4a P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
W20240503 02:14:37.868813 502 leader_election.cc:334] T 0ec95509de3443bd96bf91a08ef7df4a P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:37.869129 29874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ec95509de3443bd96bf91a08ef7df4a" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:14:37.869753 504 leader_election.cc:302] T 0ec95509de3443bd96bf91a08ef7df4a P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f575ce92c0224eeb80b9fae5f6561fbe; no voters: 16440e365430405fb10366ab70e2b429, 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:14:37.870170 14545 raft_consensus.cc:2726] T 0ec95509de3443bd96bf91a08ef7df4a P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:14:37.933465 29985 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:14:37.938601 14332 raft_consensus.cc:482] T e8a76caac93d441b92e65380bf6dc03e P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:37.938794 14332 raft_consensus.cc:504] T e8a76caac93d441b92e65380bf6dc03e P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:37.939515 14332 leader_election.cc:288] T e8a76caac93d441b92e65380bf6dc03e P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:14:37.940723 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a76caac93d441b92e65380bf6dc03e" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
W20240503 02:14:37.941641 371 leader_election.cc:334] T e8a76caac93d441b92e65380bf6dc03e P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:37.949393 370 leader_election.cc:302] T e8a76caac93d441b92e65380bf6dc03e P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 9a4d92bc25374fb3960fde7d10be309d, a9094240860e43bfb129193f3bfd5be6
I20240503 02:14:37.949666 14332 raft_consensus.cc:2726] T e8a76caac93d441b92e65380bf6dc03e P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:14:37.956861 14545 raft_consensus.cc:482] T 9cd1418eb38d420c9c41d660476eafb0 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:37.957100 14545 raft_consensus.cc:504] T 9cd1418eb38d420c9c41d660476eafb0 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:14:37.957892 14545 leader_election.cc:288] T 9cd1418eb38d420c9c41d660476eafb0 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:14:37.958537 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd1418eb38d420c9c41d660476eafb0" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:14:37.958745 30004 raft_consensus.cc:2454] T 9cd1418eb38d420c9c41d660476eafb0 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 1.
W20240503 02:14:37.960691 502 leader_election.cc:334] T 9cd1418eb38d420c9c41d660476eafb0 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:37.961392 14202 raft_consensus.cc:482] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:37.961359 29874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd1418eb38d420c9c41d660476eafb0" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:14:37.961542 14202 raft_consensus.cc:504] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:37.961611 30244 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:37.961922 504 leader_election.cc:302] T 9cd1418eb38d420c9c41d660476eafb0 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, f575ce92c0224eeb80b9fae5f6561fbe; no voters: 16440e365430405fb10366ab70e2b429, 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:14:37.962046 14202 leader_election.cc:288] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:14:37.962221 13720 raft_consensus.cc:2726] T 9cd1418eb38d420c9c41d660476eafb0 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:14:37.962821 435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a76caac93d441b92e65380bf6dc03e" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:14:37.963027 435 raft_consensus.cc:2454] T e8a76caac93d441b92e65380bf6dc03e P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 1.
I20240503 02:14:37.963958 30070 leader_election.cc:302] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:14:37.964984 14202 raft_consensus.cc:2781] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:14:37.965091 14202 raft_consensus.cc:482] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:37.965174 14202 raft_consensus.cc:3037] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:37.967489 29853 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240503 02:14:37.968870 30070 leader_election.cc:334] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:37.969672 547 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:14:37.972486 14202 raft_consensus.cc:504] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:37.977710 14202 leader_election.cc:288] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 election: Requested vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:14:37.978277 435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a76caac93d441b92e65380bf6dc03e" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:14:37.978474 435 raft_consensus.cc:3037] T e8a76caac93d441b92e65380bf6dc03e P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:14:37.979403 30070 leader_election.cc:334] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:37.982005 435 raft_consensus.cc:2454] T e8a76caac93d441b92e65380bf6dc03e P fbbc2be6382e47908ecb7c68c367bd26 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 2.
I20240503 02:14:37.983973 30070 leader_election.cc:302] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6, fbbc2be6382e47908ecb7c68c367bd26; no voters: 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:14:37.984416 14202 raft_consensus.cc:2781] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:14:37.987668 30114 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:37.995832 13959 raft_consensus.cc:482] T 58e1aaecf47944569296fc1e0780efb5 P fbbc2be6382e47908ecb7c68c367bd26 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:37.995992 13959 raft_consensus.cc:504] T 58e1aaecf47944569296fc1e0780efb5 P fbbc2be6382e47908ecb7c68c367bd26 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:14:37.996701 13959 leader_election.cc:288] T 58e1aaecf47944569296fc1e0780efb5 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:14:37.997192 565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58e1aaecf47944569296fc1e0780efb5" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:14:37.998445 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58e1aaecf47944569296fc1e0780efb5" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
W20240503 02:14:37.998847 371 leader_election.cc:334] T 58e1aaecf47944569296fc1e0780efb5 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:37.998965 371 leader_election.cc:302] T 58e1aaecf47944569296fc1e0780efb5 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:14:37.999609 13959 raft_consensus.cc:2726] T 58e1aaecf47944569296fc1e0780efb5 P fbbc2be6382e47908ecb7c68c367bd26 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:14:38.012459 32755 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:38.027436 13720 raft_consensus.cc:482] T 58e1aaecf47944569296fc1e0780efb5 P f575ce92c0224eeb80b9fae5f6561fbe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.027621 13720 raft_consensus.cc:504] T 58e1aaecf47944569296fc1e0780efb5 P f575ce92c0224eeb80b9fae5f6561fbe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:14:38.029456 13720 leader_election.cc:288] T 58e1aaecf47944569296fc1e0780efb5 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:14:38.031960 435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58e1aaecf47944569296fc1e0780efb5" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:14:38.032182 435 raft_consensus.cc:2454] T 58e1aaecf47944569296fc1e0780efb5 P fbbc2be6382e47908ecb7c68c367bd26 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 2.
I20240503 02:14:38.032779 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58e1aaecf47944569296fc1e0780efb5" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
W20240503 02:14:38.034827 502 leader_election.cc:334] T 58e1aaecf47944569296fc1e0780efb5 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.034957 502 leader_election.cc:302] T 58e1aaecf47944569296fc1e0780efb5 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: f575ce92c0224eeb80b9fae5f6561fbe, fbbc2be6382e47908ecb7c68c367bd26; no voters: 1f10f80f13c84e80b03eda7085a3b012, 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:14:38.035480 13720 raft_consensus.cc:2726] T 58e1aaecf47944569296fc1e0780efb5 P f575ce92c0224eeb80b9fae5f6561fbe [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:14:38.038906 14202 raft_consensus.cc:686] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 [term 2 LEADER]: Becoming Leader. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Running, Role: LEADER
I20240503 02:14:38.039188 14202 consensus_queue.cc:237] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:38.042512 415 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:14:38.065008 29734 catalog_manager.cc:5484] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 reported cstate change: term changed from 1 to 2, leader changed from 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) to a9094240860e43bfb129193f3bfd5be6 (127.22.108.3). New cstate: current_term: 2 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:38.069610 14008 raft_consensus.cc:482] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.069797 14008 raft_consensus.cc:504] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:14:38.076171 14008 leader_election.cc:288] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:14:38.076562 565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd1418eb38d420c9c41d660476eafb0" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:14:38.079741 13959 raft_consensus.cc:482] T 28f3f05f0a344cd683cb9e010273f908 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.079880 13959 raft_consensus.cc:504] T 28f3f05f0a344cd683cb9e010273f908 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:14:38.080430 13959 leader_election.cc:288] T 28f3f05f0a344cd683cb9e010273f908 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:14:38.082523 29874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28f3f05f0a344cd683cb9e010273f908" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:14:38.083740 565 raft_consensus.cc:2454] T 9cd1418eb38d420c9c41d660476eafb0 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:14:38.088590 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd1418eb38d420c9c41d660476eafb0" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:14:38.088812 30004 raft_consensus.cc:2454] T 9cd1418eb38d420c9c41d660476eafb0 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:14:38.090183 14505 raft_consensus.cc:482] T 9b3b9d83e4c742bfa62624676cbb6a11 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.090348 14505 raft_consensus.cc:504] T 9b3b9d83e4c742bfa62624676cbb6a11 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:38.091187 14505 leader_election.cc:288] T 9b3b9d83e4c742bfa62624676cbb6a11 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
W20240503 02:14:38.093278 32709 leader_election.cc:334] T 9b3b9d83e4c742bfa62624676cbb6a11 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.093577 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b3b9d83e4c742bfa62624676cbb6a11" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:14:38.094182 32708 leader_election.cc:302] T 9b3b9d83e4c742bfa62624676cbb6a11 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead; no voters: 9a4d92bc25374fb3960fde7d10be309d, a9094240860e43bfb129193f3bfd5be6
I20240503 02:14:38.096033 14505 raft_consensus.cc:2726] T 9b3b9d83e4c742bfa62624676cbb6a11 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:14:38.098009 29809 leader_election.cc:302] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:14:38.098284 14008 raft_consensus.cc:2781] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:14:38.098385 14008 raft_consensus.cc:482] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.098472 14008 raft_consensus.cc:3037] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:14:38.100026 29809 leader_election.cc:334] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.100504 14505 raft_consensus.cc:482] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.100636 14505 raft_consensus.cc:504] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:38.101231 14505 leader_election.cc:288] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:14:38.102258 565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec091fa0ba947d68577d562e538ab56" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
W20240503 02:14:38.103076 32709 leader_election.cc:334] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.104341 32708 leader_election.cc:302] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:14:38.104892 14505 raft_consensus.cc:2726] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:14:38.105197 371 leader_election.cc:334] T 28f3f05f0a344cd683cb9e010273f908 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.105342 371 leader_election.cc:302] T 28f3f05f0a344cd683cb9e010273f908 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 16440e365430405fb10366ab70e2b429, 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:14:38.105881 13959 raft_consensus.cc:2726] T 28f3f05f0a344cd683cb9e010273f908 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:14:38.105880 13720 raft_consensus.cc:482] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.106029 13720 raft_consensus.cc:504] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:38.105594 14008 raft_consensus.cc:504] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:14:38.107082 14008 leader_election.cc:288] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 election: Requested vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:14:38.107290 565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd1418eb38d420c9c41d660476eafb0" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:14:38.107479 565 raft_consensus.cc:3037] T 9cd1418eb38d420c9c41d660476eafb0 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:38.112880 565 raft_consensus.cc:2454] T 9cd1418eb38d420c9c41d660476eafb0 P f575ce92c0224eeb80b9fae5f6561fbe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 2.
I20240503 02:14:38.115007 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd1418eb38d420c9c41d660476eafb0" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401"
I20240503 02:14:38.115204 30004 raft_consensus.cc:3037] T 9cd1418eb38d420c9c41d660476eafb0 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:38.117946 30004 raft_consensus.cc:2454] T 9cd1418eb38d420c9c41d660476eafb0 P 155735c7f8b14997ac4c770aee2a9401 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 2.
I20240503 02:14:38.118357 14202 raft_consensus.cc:482] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.118494 14202 raft_consensus.cc:504] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:38.119091 14202 leader_election.cc:288] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:14:38.119822 29809 leader_election.cc:302] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:14:38.120169 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b3b9d83e4c742bfa62624676cbb6a11" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:14:38.120390 305 raft_consensus.cc:2454] T 9b3b9d83e4c742bfa62624676cbb6a11 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 1.
W20240503 02:14:38.120764 29809 leader_election.cc:334] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.121534 13720 leader_election.cc:288] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:14:38.122174 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec091fa0ba947d68577d562e538ab56" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:14:38.122354 305 raft_consensus.cc:2454] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 1.
I20240503 02:14:38.122444 14008 raft_consensus.cc:2781] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:14:38.122582 14008 raft_consensus.cc:686] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 [term 2 LEADER]: Becoming Leader. State: Replica: 16440e365430405fb10366ab70e2b429, State: Running, Role: LEADER
I20240503 02:14:38.122871 502 leader_election.cc:302] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:14:38.123123 13720 raft_consensus.cc:2781] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:14:38.123239 13720 raft_consensus.cc:482] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.123324 13720 raft_consensus.cc:3037] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:38.125604 13720 raft_consensus.cc:504] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:38.126138 13720 leader_election.cc:288] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 2 election: Requested vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:14:38.126503 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec091fa0ba947d68577d562e538ab56" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:14:38.126667 305 raft_consensus.cc:3037] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:14:38.126860 502 leader_election.cc:334] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.129230 305 raft_consensus.cc:2454] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f575ce92c0224eeb80b9fae5f6561fbe in term 2.
I20240503 02:14:38.130327 502 leader_election.cc:302] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:14:38.130887 14545 raft_consensus.cc:2781] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:14:38.131038 14545 raft_consensus.cc:686] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe [term 2 LEADER]: Becoming Leader. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Running, Role: LEADER
I20240503 02:14:38.131294 14545 consensus_queue.cc:237] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
W20240503 02:14:38.132869 502 leader_election.cc:334] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:38.133886 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.122879 14008 consensus_queue.cc:237] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
W20240503 02:14:38.134325 30070 leader_election.cc:334] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.134691 30070 leader_election.cc:302] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, a9094240860e43bfb129193f3bfd5be6; no voters: 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:14:38.135212 14202 raft_consensus.cc:2781] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:14:38.135303 14202 raft_consensus.cc:482] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.135383 14202 raft_consensus.cc:3037] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:38.138075 14202 raft_consensus.cc:504] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:38.151463 14202 leader_election.cc:288] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 election: Requested vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:14:38.152436 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b3b9d83e4c742bfa62624676cbb6a11" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:14:38.152674 305 raft_consensus.cc:3037] T 9b3b9d83e4c742bfa62624676cbb6a11 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:14:38.160198 30070 leader_election.cc:334] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.161566 305 raft_consensus.cc:2454] T 9b3b9d83e4c742bfa62624676cbb6a11 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 2.
I20240503 02:14:38.164435 30070 leader_election.cc:302] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, a9094240860e43bfb129193f3bfd5be6; no voters: 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:14:38.165546 14202 raft_consensus.cc:2781] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:14:38.165994 14202 raft_consensus.cc:686] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 [term 2 LEADER]: Becoming Leader. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Running, Role: LEADER
I20240503 02:14:38.166366 14202 consensus_queue.cc:237] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:38.169325 14266 raft_consensus.cc:482] T 7263a253c5884bce987d28e188ccca11 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.169529 14266 raft_consensus.cc:504] T 7263a253c5884bce987d28e188ccca11 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:14:38.170327 14266 leader_election.cc:288] T 7263a253c5884bce987d28e188ccca11 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:14:38.171206 435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7263a253c5884bce987d28e188ccca11" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
W20240503 02:14:38.172721 30200 leader_election.cc:334] T 7263a253c5884bce987d28e188ccca11 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.172868 30200 leader_election.cc:302] T 7263a253c5884bce987d28e188ccca11 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012; no voters: 9a4d92bc25374fb3960fde7d10be309d, fbbc2be6382e47908ecb7c68c367bd26
I20240503 02:14:38.173441 14441 raft_consensus.cc:2726] T 7263a253c5884bce987d28e188ccca11 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:14:38.190549 29734 catalog_manager.cc:5484] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 reported cstate change: term changed from 1 to 2, leader changed from 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) to 16440e365430405fb10366ab70e2b429 (127.22.108.1). New cstate: current_term: 2 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:14:38.191608 29732 catalog_manager.cc:5484] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe reported cstate change: term changed from 1 to 2, leader changed from 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) to f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8). New cstate: current_term: 2 leader_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:38.195961 14613 raft_consensus.cc:482] T c42a89fd9daf427bb5a60b4ee76a9287 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.196117 14613 raft_consensus.cc:504] T c42a89fd9daf427bb5a60b4ee76a9287 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:14:38.196753 14613 leader_election.cc:288] T c42a89fd9daf427bb5a60b4ee76a9287 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:14:38.198740 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c42a89fd9daf427bb5a60b4ee76a9287" candidate_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
I20240503 02:14:38.199839 14505 raft_consensus.cc:482] T 20e81486fbbe44e8888fb7c5bb5b355e P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.199999 14505 raft_consensus.cc:504] T 20e81486fbbe44e8888fb7c5bb5b355e P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:14:38.200668 14505 leader_election.cc:288] T 20e81486fbbe44e8888fb7c5bb5b355e P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
W20240503 02:14:38.200947 502 leader_election.cc:334] T c42a89fd9daf427bb5a60b4ee76a9287 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.202091 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20e81486fbbe44e8888fb7c5bb5b355e" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
W20240503 02:14:38.202414 32709 leader_election.cc:334] T 20e81486fbbe44e8888fb7c5bb5b355e P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.205130 32709 leader_election.cc:302] T 20e81486fbbe44e8888fb7c5bb5b355e P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead; no voters: 155735c7f8b14997ac4c770aee2a9401, 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:14:38.205535 14505 raft_consensus.cc:2726] T 20e81486fbbe44e8888fb7c5bb5b355e P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:14:38.201233 502 leader_election.cc:302] T c42a89fd9daf427bb5a60b4ee76a9287 P f575ce92c0224eeb80b9fae5f6561fbe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f575ce92c0224eeb80b9fae5f6561fbe; no voters: 155735c7f8b14997ac4c770aee2a9401, 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:14:38.206058 14613 raft_consensus.cc:2726] T c42a89fd9daf427bb5a60b4ee76a9287 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:14:38.220006 14332 raft_consensus.cc:482] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.220155 14332 raft_consensus.cc:504] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:14:38.220326 14266 raft_consensus.cc:482] T cf0743dee35b411e8fd069322d4f3da6 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.220468 14266 raft_consensus.cc:504] T cf0743dee35b411e8fd069322d4f3da6 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:14:38.220780 14332 leader_election.cc:288] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:14:38.221226 14266 leader_election.cc:288] T cf0743dee35b411e8fd069322d4f3da6 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:14:38.223307 29734 catalog_manager.cc:5484] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 reported cstate change: term changed from 1 to 2, leader changed from 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) to a9094240860e43bfb129193f3bfd5be6 (127.22.108.3). New cstate: current_term: 2 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
W20240503 02:14:38.223948 30200 leader_election.cc:334] T cf0743dee35b411e8fd069322d4f3da6 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.225745 435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf0743dee35b411e8fd069322d4f3da6" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:14:38.227484 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7263a253c5884bce987d28e188ccca11" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:14:38.227699 30266 raft_consensus.cc:2454] T 7263a253c5884bce987d28e188ccca11 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 1.
I20240503 02:14:38.229537 371 leader_election.cc:302] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:14:38.230698 14332 raft_consensus.cc:2781] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:14:38.230782 14332 raft_consensus.cc:482] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.230844 14332 raft_consensus.cc:3037] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:38.233595 14008 raft_consensus.cc:482] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.233588 14332 raft_consensus.cc:504] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:14:38.233752 14008 raft_consensus.cc:504] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:14:38.234336 14008 leader_election.cc:288] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:14:38.235929 14441 raft_consensus.cc:482] T 3f3ec15b1ec043d1a8f252f42a584f63 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.236085 14441 raft_consensus.cc:504] T 3f3ec15b1ec043d1a8f252f42a584f63 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:14:38.236542 14441 leader_election.cc:288] T 3f3ec15b1ec043d1a8f252f42a584f63 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:14:38.236804 435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28f3f05f0a344cd683cb9e010273f908" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:14:38.237005 435 raft_consensus.cc:2454] T 28f3f05f0a344cd683cb9e010273f908 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
W20240503 02:14:38.237615 371 leader_election.cc:334] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.237811 14008 raft_consensus.cc:482] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.237957 14008 raft_consensus.cc:504] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:14:38.238585 14008 leader_election.cc:288] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:14:38.238710 14332 leader_election.cc:288] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 election: Requested vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:14:38.238813 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7263a253c5884bce987d28e188ccca11" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:14:38.239027 30266 raft_consensus.cc:3037] T 7263a253c5884bce987d28e188ccca11 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:38.238986 565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ec95509de3443bd96bf91a08ef7df4a" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:14:38.239657 565 raft_consensus.cc:2454] T 0ec95509de3443bd96bf91a08ef7df4a P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:14:38.240036 29808 leader_election.cc:302] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:14:38.240316 14008 raft_consensus.cc:2781] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:14:38.240419 14008 raft_consensus.cc:482] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.240510 14008 raft_consensus.cc:3037] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:38.240505 30200 leader_election.cc:302] T cf0743dee35b411e8fd069322d4f3da6 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012; no voters: 9a4d92bc25374fb3960fde7d10be309d, fbbc2be6382e47908ecb7c68c367bd26
I20240503 02:14:38.240777 14266 raft_consensus.cc:2726] T cf0743dee35b411e8fd069322d4f3da6 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:14:38.241753 29809 leader_election.cc:302] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:14:38.241806 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3ec15b1ec043d1a8f252f42a584f63" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:14:38.242272 13881 raft_consensus.cc:2781] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:14:38.242358 13881 raft_consensus.cc:482] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.242451 13881 raft_consensus.cc:3037] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:14:38.243760 371 leader_election.cc:334] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.246239 30266 raft_consensus.cc:2454] T 7263a253c5884bce987d28e188ccca11 P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 2.
I20240503 02:14:38.246753 371 leader_election.cc:302] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, fbbc2be6382e47908ecb7c68c367bd26; no voters: 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:14:38.246683 13881 raft_consensus.cc:504] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:14:38.247116 14332 raft_consensus.cc:2781] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:14:38.247231 14332 raft_consensus.cc:686] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 [term 2 LEADER]: Becoming Leader. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Running, Role: LEADER
I20240503 02:14:38.247284 13881 leader_election.cc:288] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 election: Requested vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:14:38.247486 14332 consensus_queue.cc:237] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:14:38.249277 14008 raft_consensus.cc:504] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:14:38.249985 14008 leader_election.cc:288] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 election: Requested vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:14:38.250509 565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ec95509de3443bd96bf91a08ef7df4a" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:14:38.250720 565 raft_consensus.cc:3037] T 0ec95509de3443bd96bf91a08ef7df4a P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:14:38.251533 29809 leader_election.cc:334] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.252913 435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28f3f05f0a344cd683cb9e010273f908" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:14:38.253235 435 raft_consensus.cc:3037] T 28f3f05f0a344cd683cb9e010273f908 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:38.253656 565 raft_consensus.cc:2454] T 0ec95509de3443bd96bf91a08ef7df4a P f575ce92c0224eeb80b9fae5f6561fbe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 2.
I20240503 02:14:38.254036 29808 leader_election.cc:302] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:14:38.254720 14608 raft_consensus.cc:2781] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:14:38.254855 14608 raft_consensus.cc:686] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 [term 2 LEADER]: Becoming Leader. State: Replica: 16440e365430405fb10366ab70e2b429, State: Running, Role: LEADER
I20240503 02:14:38.255095 14608 consensus_queue.cc:237] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:14:38.256552 435 raft_consensus.cc:2454] T 28f3f05f0a344cd683cb9e010273f908 P fbbc2be6382e47908ecb7c68c367bd26 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 2.
W20240503 02:14:38.257139 29809 leader_election.cc:334] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.259261 14266 raft_consensus.cc:482] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.259428 14266 raft_consensus.cc:504] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:14:38.259855 29809 leader_election.cc:302] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:14:38.260670 14266 leader_election.cc:288] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:14:38.261211 14008 raft_consensus.cc:2781] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:14:38.261400 14008 raft_consensus.cc:686] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 [term 2 LEADER]: Becoming Leader. State: Replica: 16440e365430405fb10366ab70e2b429, State: Running, Role: LEADER
W20240503 02:14:38.261421 30200 leader_election.cc:334] T 3f3ec15b1ec043d1a8f252f42a584f63 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.261534 30200 leader_election.cc:302] T 3f3ec15b1ec043d1a8f252f42a584f63 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012; no voters: 8a666e798fe8477d8dd1e51b385d5ead, 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:14:38.261840 14008 consensus_queue.cc:237] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:14:38.264387 29734 catalog_manager.cc:5484] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: term changed from 1 to 2, leader changed from 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) to fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7). New cstate: current_term: 2 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:38.266731 14266 raft_consensus.cc:2726] T 3f3ec15b1ec043d1a8f252f42a584f63 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:14:38.267580 435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58e1aaecf47944569296fc1e0780efb5" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:14:38.267822 435 raft_consensus.cc:2454] T 58e1aaecf47944569296fc1e0780efb5 P fbbc2be6382e47908ecb7c68c367bd26 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 2.
W20240503 02:14:38.270177 30200 leader_election.cc:334] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.275875 565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58e1aaecf47944569296fc1e0780efb5" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:14:38.276158 565 raft_consensus.cc:2454] T 58e1aaecf47944569296fc1e0780efb5 P f575ce92c0224eeb80b9fae5f6561fbe [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 2.
I20240503 02:14:38.276810 30199 leader_election.cc:302] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, f575ce92c0224eeb80b9fae5f6561fbe, fbbc2be6382e47908ecb7c68c367bd26; no voters: 9a4d92bc25374fb3960fde7d10be309d
W20240503 02:14:38.278182 29809 leader_election.cc:334] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.278427 14266 raft_consensus.cc:2781] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20240503 02:14:38.278446 29809 leader_election.cc:334] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.278558 14266 raft_consensus.cc:482] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.279479 14266 raft_consensus.cc:3037] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:14:38.282800 14046 raft_consensus.cc:482] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.282666 14266 raft_consensus.cc:504] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:14:38.282984 14046 raft_consensus.cc:504] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:14:38.283847 14046 leader_election.cc:288] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:14:38.285408 29734 catalog_manager.cc:5484] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 reported cstate change: term changed from 1 to 2, leader changed from 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) to 16440e365430405fb10366ab70e2b429 (127.22.108.1). New cstate: current_term: 2 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:38.291680 13959 raft_consensus.cc:482] T 16cf978ac35341d4adea71a17a91194c P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.291865 13959 raft_consensus.cc:504] T 16cf978ac35341d4adea71a17a91194c P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:14:38.292657 13959 leader_election.cc:288] T 16cf978ac35341d4adea71a17a91194c P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:14:38.292690 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c42a89fd9daf427bb5a60b4ee76a9287" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" is_pre_election: true
I20240503 02:14:38.292905 567 raft_consensus.cc:2454] T c42a89fd9daf427bb5a60b4ee76a9287 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:14:38.293982 29939 leader_election.cc:302] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:14:38.294315 14046 raft_consensus.cc:2781] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:14:38.294409 14046 raft_consensus.cc:482] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.294502 14046 raft_consensus.cc:3037] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:38.295897 29874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16cf978ac35341d4adea71a17a91194c" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:14:38.297135 434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58e1aaecf47944569296fc1e0780efb5" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:14:38.297362 434 raft_consensus.cc:3037] T 58e1aaecf47944569296fc1e0780efb5 P fbbc2be6382e47908ecb7c68c367bd26 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:14:38.298224 565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58e1aaecf47944569296fc1e0780efb5" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:14:38.298457 565 raft_consensus.cc:3037] T 58e1aaecf47944569296fc1e0780efb5 P f575ce92c0224eeb80b9fae5f6561fbe [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:14:38.298871 14046 raft_consensus.cc:504] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
W20240503 02:14:38.299090 371 leader_election.cc:334] T 16cf978ac35341d4adea71a17a91194c P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.299212 371 leader_election.cc:302] T 16cf978ac35341d4adea71a17a91194c P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 16440e365430405fb10366ab70e2b429, 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:14:38.299589 14332 raft_consensus.cc:2726] T 16cf978ac35341d4adea71a17a91194c P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:14:38.300050 434 raft_consensus.cc:2454] T 58e1aaecf47944569296fc1e0780efb5 P fbbc2be6382e47908ecb7c68c367bd26 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 3.
I20240503 02:14:38.300593 567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c42a89fd9daf427bb5a60b4ee76a9287" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f575ce92c0224eeb80b9fae5f6561fbe"
I20240503 02:14:38.300773 567 raft_consensus.cc:3037] T c42a89fd9daf427bb5a60b4ee76a9287 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:14:38.302197 30200 leader_election.cc:334] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.303325 567 raft_consensus.cc:2454] T c42a89fd9daf427bb5a60b4ee76a9287 P f575ce92c0224eeb80b9fae5f6561fbe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 2.
I20240503 02:14:38.303855 29939 leader_election.cc:302] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, f575ce92c0224eeb80b9fae5f6561fbe; no voters:
I20240503 02:14:38.304342 14547 raft_consensus.cc:2781] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:14:38.305104 14046 leader_election.cc:288] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 2 election: Requested vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:14:38.306862 565 raft_consensus.cc:2454] T 58e1aaecf47944569296fc1e0780efb5 P f575ce92c0224eeb80b9fae5f6561fbe [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 3.
W20240503 02:14:38.307399 29940 leader_election.cc:334] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.307569 30199 leader_election.cc:302] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, f575ce92c0224eeb80b9fae5f6561fbe, fbbc2be6382e47908ecb7c68c367bd26; no voters: 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:14:38.308264 14441 raft_consensus.cc:2781] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:14:38.308421 14441 raft_consensus.cc:686] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 [term 3 LEADER]: Becoming Leader. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Running, Role: LEADER
I20240503 02:14:38.308732 14441 consensus_queue.cc:237] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
W20240503 02:14:38.314941 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:38.315071 29940 leader_election.cc:334] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.316210 14266 leader_election.cc:288] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 3 election: Requested vote from peers fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:14:38.317601 14208 raft_consensus.cc:482] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.317793 14208 raft_consensus.cc:504] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:14:38.318717 14208 leader_election.cc:288] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:14:38.320282 32709 leader_election.cc:334] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.329437 14202 raft_consensus.cc:482] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.329609 14202 raft_consensus.cc:504] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:38.330017 29734 catalog_manager.cc:5484] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 reported cstate change: term changed from 1 to 2, leader changed from 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) to 16440e365430405fb10366ab70e2b429 (127.22.108.1). New cstate: current_term: 2 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:38.330333 14202 leader_election.cc:288] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:14:38.331568 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3ec15b1ec043d1a8f252f42a584f63" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:14:38.331779 30266 raft_consensus.cc:2454] T 3f3ec15b1ec043d1a8f252f42a584f63 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
W20240503 02:14:38.332494 30070 leader_election.cc:334] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.333657 32709 leader_election.cc:302] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, 8a666e798fe8477d8dd1e51b385d5ead; no voters: 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:14:38.334136 14208 raft_consensus.cc:2781] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:14:38.334242 14208 raft_consensus.cc:482] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.334322 14208 raft_consensus.cc:3037] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:38.336920 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a132e4747824d5989858e649bc243c2" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:14:38.337179 30266 raft_consensus.cc:2454] T 2a132e4747824d5989858e649bc243c2 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 1.
I20240503 02:14:38.337610 30070 leader_election.cc:302] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, a9094240860e43bfb129193f3bfd5be6; no voters: 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:14:38.338133 14202 raft_consensus.cc:2781] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:14:38.338038 14208 raft_consensus.cc:504] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:14:38.338235 14202 raft_consensus.cc:482] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.338318 14202 raft_consensus.cc:3037] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:38.340696 14505 raft_consensus.cc:482] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.340842 14505 raft_consensus.cc:504] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:38.341516 14505 leader_election.cc:288] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
W20240503 02:14:38.341760 32709 leader_election.cc:334] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.341816 14632 raft_consensus.cc:482] T 2a132e4747824d5989858e649bc243c2 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.341962 14632 raft_consensus.cc:504] T 2a132e4747824d5989858e649bc243c2 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:38.341980 14202 raft_consensus.cc:504] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:38.342572 14632 leader_election.cc:288] T 2a132e4747824d5989858e649bc243c2 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:14:38.343329 14202 leader_election.cc:288] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 election: Requested vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:14:38.343547 32709 leader_election.cc:334] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.343751 14208 leader_election.cc:288] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 election: Requested vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:14:38.344713 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a132e4747824d5989858e649bc243c2" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:14:38.345021 30134 raft_consensus.cc:2379] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1f10f80f13c84e80b03eda7085a3b012 in current term 2: Already voted for candidate a9094240860e43bfb129193f3bfd5be6 in this term.
W20240503 02:14:38.345510 30070 leader_election.cc:334] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.348281 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a132e4747824d5989858e649bc243c2" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:14:38.348621 30266 raft_consensus.cc:3037] T 2a132e4747824d5989858e649bc243c2 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:38.353148 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3ec15b1ec043d1a8f252f42a584f63" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:14:38.353228 30266 raft_consensus.cc:2454] T 2a132e4747824d5989858e649bc243c2 P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 2.
I20240503 02:14:38.353365 30265 raft_consensus.cc:3037] T 3f3ec15b1ec043d1a8f252f42a584f63 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:38.354714 30070 leader_election.cc:302] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, a9094240860e43bfb129193f3bfd5be6; no voters: 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:14:38.355907 14202 raft_consensus.cc:2781] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:14:38.356487 30265 raft_consensus.cc:2454] T 3f3ec15b1ec043d1a8f252f42a584f63 P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 2.
I20240503 02:14:38.356513 14202 raft_consensus.cc:686] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 [term 2 LEADER]: Becoming Leader. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Running, Role: LEADER
I20240503 02:14:38.356864 14202 consensus_queue.cc:237] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:38.358028 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8fcfa3f3ab46b182b692a0c4fc61ed" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:14:38.359418 32709 leader_election.cc:302] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, 8a666e798fe8477d8dd1e51b385d5ead; no voters: 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:14:38.360023 14208 raft_consensus.cc:2781] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:14:38.360333 14208 raft_consensus.cc:686] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 LEADER]: Becoming Leader. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Running, Role: LEADER
I20240503 02:14:38.360803 14208 consensus_queue.cc:237] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:14:38.362571 29732 catalog_manager.cc:5484] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 reported cstate change: term changed from 1 to 2, leader changed from 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) to a9094240860e43bfb129193f3bfd5be6 (127.22.108.3). New cstate: current_term: 2 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:38.364570 14547 raft_consensus.cc:686] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 [term 2 LEADER]: Becoming Leader. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Running, Role: LEADER
I20240503 02:14:38.365536 14547 consensus_queue.cc:237] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:14:38.367208 32709 leader_election.cc:302] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead; no voters: 1f10f80f13c84e80b03eda7085a3b012, 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:14:38.367434 14505 raft_consensus.cc:2726] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:14:38.371524 30200 leader_election.cc:334] T 2a132e4747824d5989858e649bc243c2 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.371737 30200 leader_election.cc:302] T 2a132e4747824d5989858e649bc243c2 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012; no voters: 9a4d92bc25374fb3960fde7d10be309d, a9094240860e43bfb129193f3bfd5be6
I20240503 02:14:38.372108 14266 raft_consensus.cc:2726] T 2a132e4747824d5989858e649bc243c2 P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:14:38.373365 29732 catalog_manager.cc:5484] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: term changed from 1 to 2, leader changed from 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) to 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6). New cstate: current_term: 2 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } }
I20240503 02:14:38.376986 29734 catalog_manager.cc:5484] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: term changed from 2 to 3, leader changed from 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) to 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4). New cstate: current_term: 3 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:38.376984 29729 catalog_manager.cc:5484] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: term changed from 1 to 2, leader changed from 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) to 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2). New cstate: current_term: 2 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:38.399328 14332 raft_consensus.cc:482] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.399580 14332 raft_consensus.cc:504] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:14:38.400372 14332 leader_election.cc:288] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:14:38.400638 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf0743dee35b411e8fd069322d4f3da6" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:14:38.400867 30265 raft_consensus.cc:2454] T cf0743dee35b411e8fd069322d4f3da6 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 1.
I20240503 02:14:38.402083 371 leader_election.cc:302] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, fbbc2be6382e47908ecb7c68c367bd26; no voters:
W20240503 02:14:38.402776 371 leader_election.cc:334] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.403259 14332 raft_consensus.cc:2781] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:14:38.403383 14332 raft_consensus.cc:482] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.403491 14332 raft_consensus.cc:3037] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:38.407174 14624 raft_consensus.cc:482] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.407294 14332 raft_consensus.cc:504] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:14:38.407392 14624 raft_consensus.cc:504] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:14:38.408241 14624 leader_election.cc:288] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:14:38.410800 434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16cf978ac35341d4adea71a17a91194c" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:14:38.410987 434 raft_consensus.cc:2454] T 16cf978ac35341d4adea71a17a91194c P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 1.
I20240503 02:14:38.412079 29809 leader_election.cc:302] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:14:38.413574 14624 raft_consensus.cc:2781] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:14:38.413674 14624 raft_consensus.cc:482] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.413761 14624 raft_consensus.cc:3037] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:38.414783 14633 raft_consensus.cc:482] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.414991 14633 raft_consensus.cc:504] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:38.415877 14633 leader_election.cc:288] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:14:38.416697 14624 raft_consensus.cc:504] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:14:38.417479 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8fcfa3f3ab46b182b692a0c4fc61ed" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:14:38.417773 305 raft_consensus.cc:2454] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 1.
W20240503 02:14:38.419123 29809 leader_election.cc:334] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.420432 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf0743dee35b411e8fd069322d4f3da6" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:14:38.420676 30265 raft_consensus.cc:3037] T cf0743dee35b411e8fd069322d4f3da6 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:38.423914 14627 raft_consensus.cc:482] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.424144 14627 raft_consensus.cc:504] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:14:38.425052 14627 leader_election.cc:288] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:14:38.426381 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20e81486fbbe44e8888fb7c5bb5b355e" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:14:38.426589 305 raft_consensus.cc:2454] T 20e81486fbbe44e8888fb7c5bb5b355e P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
I20240503 02:14:38.427171 29940 leader_election.cc:302] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 8a666e798fe8477d8dd1e51b385d5ead; no voters:
I20240503 02:14:38.427649 14547 raft_consensus.cc:2781] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:14:38.427737 14547 raft_consensus.cc:482] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.427829 14547 raft_consensus.cc:3037] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:38.431154 14547 raft_consensus.cc:504] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:14:38.431834 14547 leader_election.cc:288] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 2 election: Requested vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
W20240503 02:14:38.433954 29940 leader_election.cc:334] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.436460 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20e81486fbbe44e8888fb7c5bb5b355e" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:14:38.436720 305 raft_consensus.cc:3037] T 20e81486fbbe44e8888fb7c5bb5b355e P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:14:38.437440 29940 leader_election.cc:334] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.437547 14624 leader_election.cc:288] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 election: Requested vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:14:38.438529 434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16cf978ac35341d4adea71a17a91194c" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:14:38.438743 434 raft_consensus.cc:3037] T 16cf978ac35341d4adea71a17a91194c P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:38.432945 30200 leader_election.cc:302] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, 8a666e798fe8477d8dd1e51b385d5ead; no voters:
W20240503 02:14:38.443854 29809 leader_election.cc:334] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.444967 434 raft_consensus.cc:2454] T 16cf978ac35341d4adea71a17a91194c P fbbc2be6382e47908ecb7c68c367bd26 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 2.
I20240503 02:14:38.445935 29809 leader_election.cc:302] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16440e365430405fb10366ab70e2b429, fbbc2be6382e47908ecb7c68c367bd26; no voters: 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:14:38.446234 14608 raft_consensus.cc:2781] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:14:38.446362 14332 leader_election.cc:288] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 election: Requested vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:14:38.446398 14608 raft_consensus.cc:686] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 [term 2 LEADER]: Becoming Leader. State: Replica: 16440e365430405fb10366ab70e2b429, State: Running, Role: LEADER
I20240503 02:14:38.445987 305 raft_consensus.cc:2454] T 20e81486fbbe44e8888fb7c5bb5b355e P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 2.
I20240503 02:14:38.446712 14608 consensus_queue.cc:237] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:14:38.447175 14266 raft_consensus.cc:2781] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240503 02:14:38.447324 30200 leader_election.cc:334] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.447950 29940 leader_election.cc:302] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, 8a666e798fe8477d8dd1e51b385d5ead; no voters: 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:14:38.448122 14266 raft_consensus.cc:482] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9a4d92bc25374fb3960fde7d10be309d)
I20240503 02:14:38.448426 14627 raft_consensus.cc:2781] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:14:38.448554 14627 raft_consensus.cc:686] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 [term 2 LEADER]: Becoming Leader. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Running, Role: LEADER
I20240503 02:14:38.448791 14627 consensus_queue.cc:237] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:14:38.449187 14266 raft_consensus.cc:3037] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:14:38.456038 371 leader_election.cc:334] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.458644 29734 catalog_manager.cc:5484] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: term changed from 1 to 2, leader changed from 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) to 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2). New cstate: current_term: 2 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:14:38.470883 29734 catalog_manager.cc:5484] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 reported cstate change: term changed from 1 to 2, leader changed from 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) to 16440e365430405fb10366ab70e2b429 (127.22.108.1). New cstate: current_term: 2 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } }
I20240503 02:14:38.475628 14266 raft_consensus.cc:504] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:38.476361 14266 leader_election.cc:288] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 2 election: Requested vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
W20240503 02:14:38.478183 30200 leader_election.cc:334] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.479131 306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8fcfa3f3ab46b182b692a0c4fc61ed" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead"
I20240503 02:14:38.479358 306 raft_consensus.cc:3037] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:38.483034 306 raft_consensus.cc:2454] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f10f80f13c84e80b03eda7085a3b012 in term 2.
I20240503 02:14:38.483709 30200 leader_election.cc:302] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, 8a666e798fe8477d8dd1e51b385d5ead; no voters: 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:14:38.484128 14266 raft_consensus.cc:2781] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:14:38.484272 14266 raft_consensus.cc:686] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 [term 2 LEADER]: Becoming Leader. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Running, Role: LEADER
I20240503 02:14:38.484580 14266 consensus_queue.cc:237] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:38.489058 29734 catalog_manager.cc:5484] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: term changed from 1 to 2, leader changed from 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) to 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4). New cstate: current_term: 2 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:38.498239 434 raft_consensus.cc:1261] T e8a76caac93d441b92e65380bf6dc03e P fbbc2be6382e47908ecb7c68c367bd26 [term 2 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:14:38.499146 14202 consensus_queue.cc:1035] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:14:38.511947 30265 raft_consensus.cc:2454] T cf0743dee35b411e8fd069322d4f3da6 P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 2.
I20240503 02:14:38.512977 371 leader_election.cc:302] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, fbbc2be6382e47908ecb7c68c367bd26; no voters: 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:14:38.513574 14332 raft_consensus.cc:2781] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:14:38.513743 14332 raft_consensus.cc:686] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 [term 2 LEADER]: Becoming Leader. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Running, Role: LEADER
I20240503 02:14:38.514035 14332 consensus_queue.cc:237] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:14:38.526602 29734 catalog_manager.cc:5484] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: term changed from 1 to 2, leader changed from 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5) to fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7). New cstate: current_term: 2 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
W20240503 02:14:38.542946 30070 consensus_peers.cc:485] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:14:38.555650 29809 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:38.560329 30004 raft_consensus.cc:1261] T 9cd1418eb38d420c9c41d660476eafb0 P 155735c7f8b14997ac4c770aee2a9401 [term 2 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:14:38.560953 14608 consensus_queue.cc:1035] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:38.562655 434 consensus_queue.cc:237] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:38.564778 30134 raft_consensus.cc:1261] T 08e4d119d90f4146ba53a1b18efe2386 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:14:38.564968 29874 raft_consensus.cc:1261] T 08e4d119d90f4146ba53a1b18efe2386 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:14:38.565454 13959 consensus_queue.cc:1035] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:14:38.565706 14332 consensus_queue.cc:1035] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:14:38.568323 13959 raft_consensus.cc:2932] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:38.568871 29873 raft_consensus.cc:2932] T 08e4d119d90f4146ba53a1b18efe2386 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:38.575572 29734 catalog_manager.cc:5484] T 08e4d119d90f4146ba53a1b18efe2386 P 16440e365430405fb10366ab70e2b429 reported cstate change: config changed from index 4 to 5, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:38.576231 30134 raft_consensus.cc:2932] T 08e4d119d90f4146ba53a1b18efe2386 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:38.579272 32755 tablet_service.cc:1514] Processing DeleteTablet for tablet 08e4d119d90f4146ba53a1b18efe2386 with delete_type TABLET_DATA_TOMBSTONED (TS 8a666e798fe8477d8dd1e51b385d5ead not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:46360
I20240503 02:14:38.580000 14664 tablet_replica.cc:330] T 08e4d119d90f4146ba53a1b18efe2386 P 8a666e798fe8477d8dd1e51b385d5ead: stopping tablet replica
I20240503 02:14:38.580266 14664 raft_consensus.cc:2229] T 08e4d119d90f4146ba53a1b18efe2386 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:38.580560 14664 raft_consensus.cc:2258] T 08e4d119d90f4146ba53a1b18efe2386 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:38.581468 14664 ts_tablet_manager.cc:1875] T 08e4d119d90f4146ba53a1b18efe2386 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 02:14:38.584138 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.589570 14664 ts_tablet_manager.cc:1888] T 08e4d119d90f4146ba53a1b18efe2386 P 8a666e798fe8477d8dd1e51b385d5ead: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:14:38.589769 14664 log.cc:1196] T 08e4d119d90f4146ba53a1b18efe2386 P 8a666e798fe8477d8dd1e51b385d5ead: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal/wals/08e4d119d90f4146ba53a1b18efe2386
I20240503 02:14:38.591042 29719 catalog_manager.cc:4830] TS 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): tablet 08e4d119d90f4146ba53a1b18efe2386 (table rebalancer_extra_table_28 [id=d70cb2e6817a47df8925321a6a778740]) successfully deleted
I20240503 02:14:38.644049 306 raft_consensus.cc:1261] T 9b3b9d83e4c742bfa62624676cbb6a11 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:14:38.644582 14606 consensus_queue.cc:1035] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:14:38.669307 30070 consensus_peers.cc:485] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:14:38.669605 502 consensus_peers.cc:485] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:14:38.677246 29809 consensus_peers.cc:485] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:38.679803 306 raft_consensus.cc:1261] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Refusing update from remote peer f575ce92c0224eeb80b9fae5f6561fbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:14:38.682096 14613 consensus_queue.cc:1035] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:14:38.688360 435 raft_consensus.cc:1261] T 28f3f05f0a344cd683cb9e010273f908 P fbbc2be6382e47908ecb7c68c367bd26 [term 2 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:14:38.696583 567 raft_consensus.cc:1261] T 58e1aaecf47944569296fc1e0780efb5 P f575ce92c0224eeb80b9fae5f6561fbe [term 3 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240503 02:14:38.698769 14624 consensus_queue.cc:1035] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:14:38.699446 371 consensus_peers.cc:485] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:38.701686 14266 consensus_queue.cc:1035] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:14:38.725490 434 raft_consensus.cc:1261] T 58e1aaecf47944569296fc1e0780efb5 P fbbc2be6382e47908ecb7c68c367bd26 [term 3 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240503 02:14:38.726250 14266 consensus_queue.cc:1035] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 02:14:38.735399 30200 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:14:38.752995 29809 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:38.758051 567 raft_consensus.cc:1261] T 9cd1418eb38d420c9c41d660476eafb0 P f575ce92c0224eeb80b9fae5f6561fbe [term 2 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:14:38.758824 14608 consensus_queue.cc:1035] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:38.781653 565 raft_consensus.cc:1261] T c42a89fd9daf427bb5a60b4ee76a9287 P f575ce92c0224eeb80b9fae5f6561fbe [term 2 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:14:38.782876 14547 consensus_queue.cc:1035] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:14:38.799726 29940 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:14:38.814707 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:38.816687 565 raft_consensus.cc:1261] T 0ec95509de3443bd96bf91a08ef7df4a P f575ce92c0224eeb80b9fae5f6561fbe [term 2 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:14:38.817674 14623 consensus_queue.cc:1035] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:14:38.820578 30265 raft_consensus.cc:1261] T 7263a253c5884bce987d28e188ccca11 P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:14:38.821269 14332 consensus_queue.cc:1035] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:38.830403 30266 raft_consensus.cc:1261] T 3f3ec15b1ec043d1a8f252f42a584f63 P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:14:38.831286 14208 consensus_queue.cc:1035] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:14:38.870406 32709 consensus_peers.cc:485] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:38.874454 30265 raft_consensus.cc:1261] T 2a132e4747824d5989858e649bc243c2 P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:14:38.875542 14606 consensus_queue.cc:1035] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:14:38.889760 30070 consensus_peers.cc:485] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:38.919791 434 raft_consensus.cc:1261] T 16cf978ac35341d4adea71a17a91194c P fbbc2be6382e47908ecb7c68c367bd26 [term 2 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:14:38.920832 14008 consensus_queue.cc:1035] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:14:38.928310 29940 consensus_peers.cc:485] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:14:38.941094 29809 consensus_peers.cc:485] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:39.012373 305 raft_consensus.cc:1261] T 20e81486fbbe44e8888fb7c5bb5b355e P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:14:39.013482 14046 consensus_queue.cc:1035] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:14:39.031172 30200 consensus_peers.cc:485] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:39.058743 29985 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 02:14:39.059072 371 consensus_peers.cc:485] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:39.061972 29853 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:39.092471 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:39.101758 30244 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:39.136915 30265 raft_consensus.cc:1261] T cf0743dee35b411e8fd069322d4f3da6 P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:14:39.137521 13959 consensus_queue.cc:1035] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:39.149581 306 raft_consensus.cc:1261] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:14:39.151648 14441 consensus_queue.cc:1035] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:14:39.222678 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:39.277567 415 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:39.279600 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:39.321391 547 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:14:39.338096 30114 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:14:39.416551 30052 maintenance_manager.cc:419] P 155735c7f8b14997ac4c770aee2a9401: Scheduling FlushMRSOp(04e47da8cf7445dcacaf1f7c8950bd34): perf score=0.033338
W20240503 02:14:39.473333 30200 consensus_peers.cc:485] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:14:39.488973 29943 maintenance_manager.cc:643] P 155735c7f8b14997ac4c770aee2a9401: FlushMRSOp(04e47da8cf7445dcacaf1f7c8950bd34) complete. Timing: real 0.072s user 0.043s sys 0.018s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":343,"dirs.run_cpu_time_us":212,"dirs.run_wall_time_us":1390,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":844,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":3629,"threads_started":2,"wal-append.queue_time_us":3285}
I20240503 02:14:39.491377 30052 maintenance_manager.cc:419] P 155735c7f8b14997ac4c770aee2a9401: Scheduling FlushMRSOp(5f4b72b138c64ac58e6477cd2b1fb215): perf score=0.033354
W20240503 02:14:39.527516 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:39.532439 30070 consensus_peers.cc:485] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:14:39.535107 29940 consensus_peers.cc:485] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:14:39.540627 29921 maintenance_manager.cc:419] P 16440e365430405fb10366ab70e2b429: Scheduling FlushMRSOp(5f4b72b138c64ac58e6477cd2b1fb215): perf score=0.033367
I20240503 02:14:39.544277 29943 maintenance_manager.cc:643] P 155735c7f8b14997ac4c770aee2a9401: FlushMRSOp(5f4b72b138c64ac58e6477cd2b1fb215) complete. Timing: real 0.053s user 0.035s sys 0.011s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":36,"dirs.run_cpu_time_us":250,"dirs.run_wall_time_us":1507,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":755,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":8742,"threads_started":1,"wal-append.queue_time_us":8629}
I20240503 02:14:39.546825 30052 maintenance_manager.cc:419] P 155735c7f8b14997ac4c770aee2a9401: Scheduling UndoDeltaBlockGCOp(5f4b72b138c64ac58e6477cd2b1fb215): 4923 bytes on disk
I20240503 02:14:39.547819 29943 maintenance_manager.cc:643] P 155735c7f8b14997ac4c770aee2a9401: UndoDeltaBlockGCOp(5f4b72b138c64ac58e6477cd2b1fb215) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20240503 02:14:39.558401 30052 maintenance_manager.cc:419] P 155735c7f8b14997ac4c770aee2a9401: Scheduling UndoDeltaBlockGCOp(04e47da8cf7445dcacaf1f7c8950bd34): 4836 bytes on disk
I20240503 02:14:39.558915 29943 maintenance_manager.cc:643] P 155735c7f8b14997ac4c770aee2a9401: UndoDeltaBlockGCOp(04e47da8cf7445dcacaf1f7c8950bd34) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20240503 02:14:39.635221 30312 maintenance_manager.cc:419] P 1f10f80f13c84e80b03eda7085a3b012: Scheduling FlushMRSOp(04e47da8cf7445dcacaf1f7c8950bd34): perf score=0.033398
I20240503 02:14:39.653486 30182 maintenance_manager.cc:419] P a9094240860e43bfb129193f3bfd5be6: Scheduling FlushMRSOp(c19017f829a74cc2b6dc41a2a25a3831): perf score=0.033392
W20240503 02:14:39.657230 371 consensus_peers.cc:485] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:14:39.669044 502 consensus_peers.cc:485] T 36d3f1edbab7477f9b7ac400160491a0 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:14:39.694454 29812 maintenance_manager.cc:643] P 16440e365430405fb10366ab70e2b429: FlushMRSOp(5f4b72b138c64ac58e6477cd2b1fb215) complete. Timing: real 0.154s user 0.062s sys 0.015s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":396,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":1742,"drs_written":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1850,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":9885,"threads_started":2,"wal-append.queue_time_us":9536}
I20240503 02:14:39.708026 29921 maintenance_manager.cc:419] P 16440e365430405fb10366ab70e2b429: Scheduling UndoDeltaBlockGCOp(5f4b72b138c64ac58e6477cd2b1fb215): 4923 bytes on disk
I20240503 02:14:39.708827 29812 maintenance_manager.cc:643] P 16440e365430405fb10366ab70e2b429: UndoDeltaBlockGCOp(5f4b72b138c64ac58e6477cd2b1fb215) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
W20240503 02:14:39.725798 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:39.729573 30200 consensus_peers.cc:485] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:14:39.741473 502 consensus_peers.cc:485] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:14:39.747839 30073 maintenance_manager.cc:643] P a9094240860e43bfb129193f3bfd5be6: FlushMRSOp(c19017f829a74cc2b6dc41a2a25a3831) complete. Timing: real 0.094s user 0.037s sys 0.013s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":15357,"dirs.run_cpu_time_us":234,"dirs.run_wall_time_us":1779,"drs_written":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4,"lbm_write_time_us":874,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":15967,"threads_started":2,"wal-append.queue_time_us":796}
W20240503 02:14:39.750167 29809 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:14:39.750298 29809 consensus_peers.cc:485] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:14:39.751281 30182 maintenance_manager.cc:419] P a9094240860e43bfb129193f3bfd5be6: Scheduling UndoDeltaBlockGCOp(c19017f829a74cc2b6dc41a2a25a3831): 4581 bytes on disk
I20240503 02:14:39.751969 30073 maintenance_manager.cc:643] P a9094240860e43bfb129193f3bfd5be6: UndoDeltaBlockGCOp(c19017f829a74cc2b6dc41a2a25a3831) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4}
I20240503 02:14:39.759625 30203 maintenance_manager.cc:643] P 1f10f80f13c84e80b03eda7085a3b012: FlushMRSOp(04e47da8cf7445dcacaf1f7c8950bd34) complete. Timing: real 0.124s user 0.045s sys 0.013s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":3908,"dirs.run_cpu_time_us":232,"dirs.run_wall_time_us":21432,"drs_written":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4,"lbm_write_time_us":978,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":4285,"threads_started":2,"wal-append.queue_time_us":351}
I20240503 02:14:39.762830 30312 maintenance_manager.cc:419] P 1f10f80f13c84e80b03eda7085a3b012: Scheduling FlushMRSOp(c19017f829a74cc2b6dc41a2a25a3831): perf score=0.033431
W20240503 02:14:39.807061 502 consensus_peers.cc:485] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:14:39.825480 30070 consensus_peers.cc:485] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:14:39.865955 30203 maintenance_manager.cc:643] P 1f10f80f13c84e80b03eda7085a3b012: FlushMRSOp(c19017f829a74cc2b6dc41a2a25a3831) complete. Timing: real 0.103s user 0.034s sys 0.030s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":73,"dirs.run_cpu_time_us":207,"dirs.run_wall_time_us":3780,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":871,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":24158,"threads_started":1,"wal-append.queue_time_us":24248}
W20240503 02:14:39.868885 371 consensus_peers.cc:485] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:14:39.874632 30312 maintenance_manager.cc:419] P 1f10f80f13c84e80b03eda7085a3b012: Scheduling UndoDeltaBlockGCOp(04e47da8cf7445dcacaf1f7c8950bd34): 4836 bytes on disk
I20240503 02:14:39.875823 30203 maintenance_manager.cc:643] P 1f10f80f13c84e80b03eda7085a3b012: UndoDeltaBlockGCOp(04e47da8cf7445dcacaf1f7c8950bd34) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":144,"lbm_reads_lt_1ms":4}
I20240503 02:14:39.879108 30312 maintenance_manager.cc:419] P 1f10f80f13c84e80b03eda7085a3b012: Scheduling UndoDeltaBlockGCOp(c19017f829a74cc2b6dc41a2a25a3831): 4581 bytes on disk
I20240503 02:14:39.879681 30203 maintenance_manager.cc:643] P 1f10f80f13c84e80b03eda7085a3b012: UndoDeltaBlockGCOp(c19017f829a74cc2b6dc41a2a25a3831) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240503 02:14:39.903249 29985 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:39.904593 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:39.905907 29853 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:39.914469 30070 consensus_peers.cc:485] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:14:39.937458 502 consensus_peers.cc:485] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:14:39.942090 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:39.952226 547 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:14:39.960815 30244 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:39.974964 30114 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 02:14:39.997429 30200 consensus_peers.cc:485] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:14:40.005514 30070 consensus_peers.cc:485] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:14:40.015677 371 consensus_peers.cc:485] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:14:40.038962 415 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:40.069219 29809 consensus_peers.cc:485] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:14:40.077210 30070 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:14:40.097446 502 consensus_peers.cc:485] T ae4373a66af14c8f8a633963dffd2261 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:14:40.098304 30200 consensus_peers.cc:485] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:14:40.104051 32709 consensus_peers.cc:485] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:14:40.179080 29940 consensus_peers.cc:485] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:14:40.205008 30070 consensus_peers.cc:485] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:14:40.207464 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:40.220408 32709 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:14:40.323345 29809 consensus_peers.cc:485] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:14:40.379999 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:40.416421 29985 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:40.439154 30244 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:40.445782 29853 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:40.462653 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:40.537389 547 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:14:40.548415 30114 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:14:40.571751 415 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:40.765631 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:40.772367 29809 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:14:40.818933 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:40.997781 29853 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:40.998090 30244 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:40.998453 29985 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:41.019328 547 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:14:41.034868 415 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:41.045320 30114 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 02:14:41.055465 502 consensus_peers.cc:485] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:14:41.077075 371 consensus_peers.cc:485] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:14:41.091279 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240503 02:14:41.096568 30070 consensus_peers.cc:485] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:14:41.153558 29809 consensus_peers.cc:485] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:14:41.161891 29809 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:14:41.166857 30070 consensus_peers.cc:485] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:14:41.180800 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:41.247535 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:41.329586 30200 consensus_peers.cc:485] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:14:41.337347 29940 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:14:41.346932 32709 consensus_peers.cc:485] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:14:41.366423 30200 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:14:41.399261 30070 consensus_peers.cc:485] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:14:41.473371 29985 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:41.481624 29921 maintenance_manager.cc:419] P 16440e365430405fb10366ab70e2b429: Scheduling FlushMRSOp(e61579e74a1a4a9dbd9a09f9b34d9c8d): perf score=0.033356
I20240503 02:14:41.486294 29853 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:41.496065 29809 consensus_peers.cc:485] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:14:41.504894 30244 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:41.524541 30182 maintenance_manager.cc:419] P a9094240860e43bfb129193f3bfd5be6: Scheduling FlushMRSOp(6bf84757d1fb44689da830a82e857787): perf score=0.033370
I20240503 02:14:41.543653 30114 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:14:41.571692 29812 maintenance_manager.cc:643] P 16440e365430405fb10366ab70e2b429: FlushMRSOp(e61579e74a1a4a9dbd9a09f9b34d9c8d) complete. Timing: real 0.090s user 0.033s sys 0.020s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":10119,"dirs.run_cpu_time_us":190,"dirs.run_wall_time_us":2272,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1069,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":6685,"threads_started":2,"wal-append.queue_time_us":313}
I20240503 02:14:41.574815 29921 maintenance_manager.cc:419] P 16440e365430405fb10366ab70e2b429: Scheduling UndoDeltaBlockGCOp(e61579e74a1a4a9dbd9a09f9b34d9c8d): 4838 bytes on disk
I20240503 02:14:41.575501 29812 maintenance_manager.cc:643] P 16440e365430405fb10366ab70e2b429: UndoDeltaBlockGCOp(e61579e74a1a4a9dbd9a09f9b34d9c8d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20240503 02:14:41.579738 30052 maintenance_manager.cc:419] P 155735c7f8b14997ac4c770aee2a9401: Scheduling FlushMRSOp(e61579e74a1a4a9dbd9a09f9b34d9c8d): perf score=0.033386
I20240503 02:14:41.606145 547 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:14:41.606580 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:41.648021 30312 maintenance_manager.cc:419] P 1f10f80f13c84e80b03eda7085a3b012: Scheduling FlushMRSOp(6bf84757d1fb44689da830a82e857787): perf score=0.033403
W20240503 02:14:41.656129 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:41.664331 371 consensus_peers.cc:485] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:14:41.670957 415 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:41.691709 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:41.735450 29940 consensus_peers.cc:485] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:14:41.742759 29943 maintenance_manager.cc:643] P 155735c7f8b14997ac4c770aee2a9401: FlushMRSOp(e61579e74a1a4a9dbd9a09f9b34d9c8d) complete. Timing: real 0.163s user 0.042s sys 0.011s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":7732,"dirs.run_cpu_time_us":225,"dirs.run_wall_time_us":12109,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":765,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":25030,"threads_started":2,"wal-append.queue_time_us":17255}
I20240503 02:14:41.745509 30052 maintenance_manager.cc:419] P 155735c7f8b14997ac4c770aee2a9401: Scheduling UndoDeltaBlockGCOp(e61579e74a1a4a9dbd9a09f9b34d9c8d): 4838 bytes on disk
I20240503 02:14:41.752831 29943 maintenance_manager.cc:643] P 155735c7f8b14997ac4c770aee2a9401: UndoDeltaBlockGCOp(e61579e74a1a4a9dbd9a09f9b34d9c8d) complete. Timing: real 0.005s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
I20240503 02:14:41.776221 30203 maintenance_manager.cc:643] P 1f10f80f13c84e80b03eda7085a3b012: FlushMRSOp(6bf84757d1fb44689da830a82e857787) complete. Timing: real 0.128s user 0.050s sys 0.007s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":25866,"dirs.run_cpu_time_us":224,"dirs.run_wall_time_us":15745,"drs_written":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4,"lbm_write_time_us":848,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":31042,"threads_started":2,"wal-append.queue_time_us":8737}
I20240503 02:14:41.778877 30312 maintenance_manager.cc:419] P 1f10f80f13c84e80b03eda7085a3b012: Scheduling FlushMRSOp(db0756f0f331402aaef0583e1f9e5318): perf score=0.033435
I20240503 02:14:41.797319 30073 maintenance_manager.cc:643] P a9094240860e43bfb129193f3bfd5be6: FlushMRSOp(6bf84757d1fb44689da830a82e857787) complete. Timing: real 0.273s user 0.059s sys 0.025s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":15967,"dirs.run_cpu_time_us":203,"dirs.run_wall_time_us":15774,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1254,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":40607,"threads_started":2,"wal-append.queue_time_us":24623}
W20240503 02:14:41.798960 30200 consensus_peers.cc:485] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:14:41.800230 30182 maintenance_manager.cc:419] P a9094240860e43bfb129193f3bfd5be6: Scheduling UndoDeltaBlockGCOp(6bf84757d1fb44689da830a82e857787): 4921 bytes on disk
I20240503 02:14:41.801004 30073 maintenance_manager.cc:643] P a9094240860e43bfb129193f3bfd5be6: UndoDeltaBlockGCOp(6bf84757d1fb44689da830a82e857787) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4}
I20240503 02:14:41.807142 30182 maintenance_manager.cc:419] P a9094240860e43bfb129193f3bfd5be6: Scheduling FlushMRSOp(db0756f0f331402aaef0583e1f9e5318): perf score=0.033443
I20240503 02:14:41.847556 30203 maintenance_manager.cc:643] P 1f10f80f13c84e80b03eda7085a3b012: FlushMRSOp(db0756f0f331402aaef0583e1f9e5318) complete. Timing: real 0.068s user 0.055s sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":37,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":1555,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":856,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":6518,"threads_started":1,"wal-append.queue_time_us":6419}
I20240503 02:14:41.851926 30312 maintenance_manager.cc:419] P 1f10f80f13c84e80b03eda7085a3b012: Scheduling UndoDeltaBlockGCOp(6bf84757d1fb44689da830a82e857787): 4921 bytes on disk
I20240503 02:14:41.853260 30203 maintenance_manager.cc:643] P 1f10f80f13c84e80b03eda7085a3b012: UndoDeltaBlockGCOp(6bf84757d1fb44689da830a82e857787) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240503 02:14:41.856546 30073 maintenance_manager.cc:643] P a9094240860e43bfb129193f3bfd5be6: FlushMRSOp(db0756f0f331402aaef0583e1f9e5318) complete. Timing: real 0.049s user 0.032s sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":172,"dirs.run_wall_time_us":1397,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":897,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":6438,"threads_started":1,"wal-append.queue_time_us":6320}
I20240503 02:14:41.860363 30182 maintenance_manager.cc:419] P a9094240860e43bfb129193f3bfd5be6: Scheduling UndoDeltaBlockGCOp(db0756f0f331402aaef0583e1f9e5318): 4579 bytes on disk
I20240503 02:14:41.861131 30073 maintenance_manager.cc:643] P a9094240860e43bfb129193f3bfd5be6: UndoDeltaBlockGCOp(db0756f0f331402aaef0583e1f9e5318) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4}
I20240503 02:14:41.870378 30312 maintenance_manager.cc:419] P 1f10f80f13c84e80b03eda7085a3b012: Scheduling UndoDeltaBlockGCOp(db0756f0f331402aaef0583e1f9e5318): 4579 bytes on disk
I20240503 02:14:41.871050 30203 maintenance_manager.cc:643] P 1f10f80f13c84e80b03eda7085a3b012: UndoDeltaBlockGCOp(db0756f0f331402aaef0583e1f9e5318) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20240503 02:14:41.918915 30135 consensus_queue.cc:237] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:14:41.922039 565 raft_consensus.cc:1261] T 88b57085de3144cea3f64922ae5ccf55 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:14:41.922048 434 raft_consensus.cc:1261] T 88b57085de3144cea3f64922ae5ccf55 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:14:41.922510 14606 consensus_queue.cc:1035] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:14:41.922762 14202 consensus_queue.cc:1035] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:14:41.925362 14202 raft_consensus.cc:2932] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:14:41.928220 434 raft_consensus.cc:2932] T 88b57085de3144cea3f64922ae5ccf55 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:14:41.930094 566 raft_consensus.cc:2932] T 88b57085de3144cea3f64922ae5ccf55 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:14:41.934506 29726 catalog_manager.cc:5484] T 88b57085de3144cea3f64922ae5ccf55 P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: config changed from index 6 to 7, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) evicted. New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:14:41.944249 29985 tablet_service.cc:1514] Processing DeleteTablet for tablet 88b57085de3144cea3f64922ae5ccf55 with delete_type TABLET_DATA_TOMBSTONED (TS 155735c7f8b14997ac4c770aee2a9401 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52158
I20240503 02:14:41.945050 15043 tablet_replica.cc:330] T 88b57085de3144cea3f64922ae5ccf55 P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:14:41.945343 15043 raft_consensus.cc:2229] T 88b57085de3144cea3f64922ae5ccf55 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:41.945730 15043 raft_consensus.cc:2258] T 88b57085de3144cea3f64922ae5ccf55 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:41.946785 15043 ts_tablet_manager.cc:1875] T 88b57085de3144cea3f64922ae5ccf55 P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:41.958104 15043 ts_tablet_manager.cc:1888] T 88b57085de3144cea3f64922ae5ccf55 P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:14:41.958431 15043 log.cc:1196] T 88b57085de3144cea3f64922ae5ccf55 P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/88b57085de3144cea3f64922ae5ccf55
I20240503 02:14:41.959355 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet 88b57085de3144cea3f64922ae5ccf55 (table rebalance_test_table_5 [id=99dbec2a2b63467ab34b96ecc8479244]) successfully deleted
W20240503 02:14:42.010890 29940 consensus_peers.cc:485] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:14:42.011721 29809 consensus_peers.cc:485] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:14:42.087581 502 consensus_peers.cc:485] T 36d3f1edbab7477f9b7ac400160491a0 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:14:42.113809 371 consensus_peers.cc:485] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:14:42.137729 30200 consensus_peers.cc:485] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:14:42.185600 371 consensus_peers.cc:485] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:14:42.193918 502 consensus_peers.cc:485] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:14:42.222796 30070 consensus_peers.cc:485] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:14:42.253813 502 consensus_peers.cc:485] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:14:42.259799 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:42.284518 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:42.326704 29985 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 02:14:42.345914 30070 consensus_peers.cc:485] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:14:42.348320 30114 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 02:14:42.389447 29809 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:14:42.400262 30244 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:42.425573 415 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:42.428064 30070 consensus_peers.cc:485] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:14:42.438091 29853 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:42.438341 547 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:14:42.439857 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240503 02:14:42.466596 371 consensus_peers.cc:485] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:14:42.469015 30070 consensus_peers.cc:485] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:14:42.538403 30200 consensus_peers.cc:485] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:14:42.543196 29940 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240503 02:14:42.554960 502 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240503 02:14:42.569885 30200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240503 02:14:42.579898 30070 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240503 02:14:42.580878 30070 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:14:42.583652 30200 consensus_peers.cc:485] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:14:42.585175 29809 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 84 similar messages]
W20240503 02:14:42.601362 32709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240503 02:14:42.603492 32709 consensus_peers.cc:485] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:14:42.612176 30004 consensus_queue.cc:237] T 8929e26a53274cecbb808dbd8b203a64 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:14:42.616437 303 raft_consensus.cc:1261] T 8929e26a53274cecbb808dbd8b203a64 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:42.617043 14627 consensus_queue.cc:1035] T 8929e26a53274cecbb808dbd8b203a64 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:42.618216 30135 raft_consensus.cc:1261] T 8929e26a53274cecbb808dbd8b203a64 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:42.619640 14627 consensus_queue.cc:1035] T 8929e26a53274cecbb808dbd8b203a64 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 02:14:42.620952 371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 55 similar messages]
I20240503 02:14:42.622176 14627 raft_consensus.cc:2932] T 8929e26a53274cecbb808dbd8b203a64 P 155735c7f8b14997ac4c770aee2a9401 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:42.625896 303 raft_consensus.cc:2932] T 8929e26a53274cecbb808dbd8b203a64 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
W20240503 02:14:42.628571 502 consensus_peers.cc:485] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:14:42.633600 29727 catalog_manager.cc:5484] T 8929e26a53274cecbb808dbd8b203a64 P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: config changed from index 2 to 3, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New cstate: current_term: 1 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:42.635654 30135 raft_consensus.cc:2932] T 8929e26a53274cecbb808dbd8b203a64 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:14:42.657333 29853 tablet_service.cc:1514] Processing DeleteTablet for tablet 8929e26a53274cecbb808dbd8b203a64 with delete_type TABLET_DATA_TOMBSTONED (TS 16440e365430405fb10366ab70e2b429 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:58744
I20240503 02:14:42.662820 15138 tablet_replica.cc:330] T 8929e26a53274cecbb808dbd8b203a64 P 16440e365430405fb10366ab70e2b429: stopping tablet replica
W20240503 02:14:42.666113 30070 consensus_peers.cc:485] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:14:42.666404 15138 raft_consensus.cc:2229] T 8929e26a53274cecbb808dbd8b203a64 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:42.666698 15138 raft_consensus.cc:2258] T 8929e26a53274cecbb808dbd8b203a64 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 02:14:42.667840 502 consensus_peers.cc:485] T ae4373a66af14c8f8a633963dffd2261 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:14:42.668596 15138 ts_tablet_manager.cc:1875] T 8929e26a53274cecbb808dbd8b203a64 P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:42.675066 15138 ts_tablet_manager.cc:1888] T 8929e26a53274cecbb808dbd8b203a64 P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:14:42.681975 15138 log.cc:1196] T 8929e26a53274cecbb808dbd8b203a64 P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/8929e26a53274cecbb808dbd8b203a64
I20240503 02:14:42.682715 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet 8929e26a53274cecbb808dbd8b203a64 (table rebalancer_extra_table_50 [id=e4dd29f93ff64f6088ff5503d1f570dd]) successfully deleted
W20240503 02:14:42.696785 29809 consensus_peers.cc:485] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:14:42.719316 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:42.851969 32709 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:14:42.913743 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:42.923478 29853 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:42.927474 29985 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:42.945492 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:42.957355 30114 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 02:14:42.960752 29809 consensus_peers.cc:485] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:14:42.963034 30244 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:42.978840 29940 consensus_peers.cc:485] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:14:43.012977 547 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 02:14:43.023738 29809 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:14:43.040591 415 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:43.318768 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:43.387065 29985 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:43.397609 29853 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:43.420599 30244 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:43.439921 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240503 02:14:43.440246 371 consensus_peers.cc:485] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:14:43.481508 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:43.496253 30070 consensus_peers.cc:485] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:14:43.501415 547 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:14:43.501936 30114 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:14:43.524818 415 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:43.531862 502 consensus_peers.cc:485] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:14:43.548053 29809 consensus_peers.cc:485] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:14:43.616603 29809 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:14:43.684705 30200 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:14:43.689152 32709 consensus_peers.cc:485] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:14:43.700682 29809 consensus_peers.cc:485] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:14:43.739235 30070 consensus_peers.cc:485] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:14:43.744427 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:43.905341 29853 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:43.908381 29985 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:43.909937 30244 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:43.915783 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:43.928462 30114 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:14:43.940016 547 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 02:14:43.972395 29940 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:14:43.983466 415 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:43.999459 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:44.032725 30070 consensus_peers.cc:485] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:14:44.076700 30200 consensus_peers.cc:485] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:14:44.095857 371 consensus_peers.cc:485] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:14:44.143886 303 consensus_queue.cc:237] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:14:44.146051 30006 raft_consensus.cc:1261] T 5cf58a111e3048449303ec45ac091b42 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:44.146549 29873 raft_consensus.cc:1261] T 5cf58a111e3048449303ec45ac091b42 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:44.146868 15078 consensus_queue.cc:1035] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:44.147569 15078 consensus_queue.cc:1035] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:44.149566 15290 raft_consensus.cc:2932] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:14:44.153443 30006 raft_consensus.cc:2932] T 5cf58a111e3048449303ec45ac091b42 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:14:44.154215 29873 raft_consensus.cc:2932] T 5cf58a111e3048449303ec45ac091b42 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:14:44.156561 29732 catalog_manager.cc:5484] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: config changed from index 2 to 3, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) evicted. New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
W20240503 02:14:44.180158 29940 consensus_peers.cc:485] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:14:44.215142 547 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cf58a111e3048449303ec45ac091b42 with delete_type TABLET_DATA_TOMBSTONED (TS f575ce92c0224eeb80b9fae5f6561fbe not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:39028
I20240503 02:14:44.215781 15330 tablet_replica.cc:330] T 5cf58a111e3048449303ec45ac091b42 P f575ce92c0224eeb80b9fae5f6561fbe: stopping tablet replica
I20240503 02:14:44.216176 15330 raft_consensus.cc:2229] T 5cf58a111e3048449303ec45ac091b42 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:44.216467 15330 raft_consensus.cc:2258] T 5cf58a111e3048449303ec45ac091b42 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:44.217881 15330 ts_tablet_manager.cc:1875] T 5cf58a111e3048449303ec45ac091b42 P f575ce92c0224eeb80b9fae5f6561fbe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 02:14:44.220268 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:44.224592 15330 ts_tablet_manager.cc:1888] T 5cf58a111e3048449303ec45ac091b42 P f575ce92c0224eeb80b9fae5f6561fbe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:14:44.224771 15330 log.cc:1196] T 5cf58a111e3048449303ec45ac091b42 P f575ce92c0224eeb80b9fae5f6561fbe: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal/wals/5cf58a111e3048449303ec45ac091b42
I20240503 02:14:44.225685 29718 catalog_manager.cc:4830] TS f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): tablet 5cf58a111e3048449303ec45ac091b42 (table rebalancer_extra_table_0 [id=c9cc90e5637140278312ab93695a8afd]) successfully deleted
W20240503 02:14:44.239151 29940 consensus_peers.cc:485] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:14:44.471856 29853 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:44.492115 30244 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:44.497603 371 consensus_peers.cc:485] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:14:44.527245 502 consensus_peers.cc:485] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:14:44.527251 29809 consensus_peers.cc:485] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:14:44.536046 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:44.582619 547 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:14:44.584440 29985 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:44.591610 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:44.625463 30114 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:14:44.659569 415 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:44.664863 30070 consensus_peers.cc:485] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:14:44.683125 30200 consensus_peers.cc:485] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:14:44.722239 371 consensus_peers.cc:485] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:14:44.733347 502 consensus_peers.cc:485] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:14:44.733634 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:44.762193 502 consensus_peers.cc:485] T 36d3f1edbab7477f9b7ac400160491a0 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:14:44.796121 30070 consensus_peers.cc:485] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:14:44.802359 30070 consensus_peers.cc:485] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:14:44.884410 30200 consensus_peers.cc:485] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:14:44.895887 30070 consensus_peers.cc:485] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:14:44.913548 29809 consensus_peers.cc:485] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:14:44.924104 30070 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:14:44.959563 32709 consensus_peers.cc:485] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:14:44.976490 30200 consensus_peers.cc:485] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:14:44.979667 29809 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:14:45.073354 371 consensus_peers.cc:485] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:14:45.082674 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:45.083613 29985 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:45.103107 29853 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:45.126168 30070 consensus_peers.cc:485] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:14:45.127628 30114 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:14:45.130342 30244 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:45.137881 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:45.145330 502 consensus_peers.cc:485] T ae4373a66af14c8f8a633963dffd2261 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:14:45.149304 547 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:14:45.155535 415 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:45.167797 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:45.292114 30200 consensus_peers.cc:485] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:14:45.394439 502 consensus_peers.cc:485] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:14:45.401397 32709 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:14:45.421356 29809 consensus_peers.cc:485] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:14:45.504187 29940 consensus_peers.cc:485] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:14:45.561647 30052 maintenance_manager.cc:419] P 155735c7f8b14997ac4c770aee2a9401: Scheduling FlushMRSOp(6635aacafed54994965ad48b2df4f163): perf score=0.033340
I20240503 02:14:45.568449 29985 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 02:14:45.578353 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:45.586727 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:45.600800 29853 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:45.633423 29921 maintenance_manager.cc:419] P 16440e365430405fb10366ab70e2b429: Scheduling FlushMRSOp(63b416f4e004490b8e38a96fd065dfea): perf score=0.033357
I20240503 02:14:45.647486 30244 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:45.667809 30312 maintenance_manager.cc:419] P 1f10f80f13c84e80b03eda7085a3b012: Scheduling FlushMRSOp(6635aacafed54994965ad48b2df4f163): perf score=0.033375
I20240503 02:14:45.668431 30182 maintenance_manager.cc:419] P a9094240860e43bfb129193f3bfd5be6: Scheduling FlushMRSOp(1c9200fa969a46e6a48c64293801382b): perf score=0.033367
I20240503 02:14:45.717171 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:45.744880 30114 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:14:45.748610 29812 maintenance_manager.cc:643] P 16440e365430405fb10366ab70e2b429: FlushMRSOp(63b416f4e004490b8e38a96fd065dfea) complete. Timing: real 0.115s user 0.035s sys 0.007s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":13226,"dirs.run_cpu_time_us":236,"dirs.run_wall_time_us":15163,"drs_written":1,"lbm_read_time_us":123,"lbm_reads_lt_1ms":4,"lbm_write_time_us":860,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":29793,"threads_started":2,"wal-append.queue_time_us":16278}
I20240503 02:14:45.751857 29921 maintenance_manager.cc:419] P 16440e365430405fb10366ab70e2b429: Scheduling UndoDeltaBlockGCOp(63b416f4e004490b8e38a96fd065dfea): 4923 bytes on disk
I20240503 02:14:45.752408 29812 maintenance_manager.cc:643] P 16440e365430405fb10366ab70e2b429: UndoDeltaBlockGCOp(63b416f4e004490b8e38a96fd065dfea) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
I20240503 02:14:45.769748 547 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:14:45.789543 415 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:45.823191 29943 maintenance_manager.cc:643] P 155735c7f8b14997ac4c770aee2a9401: FlushMRSOp(6635aacafed54994965ad48b2df4f163) complete. Timing: real 0.261s user 0.043s sys 0.007s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":18965,"dirs.run_cpu_time_us":220,"dirs.run_wall_time_us":17381,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":891,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":40305,"threads_started":2,"wal-append.queue_time_us":18946}
I20240503 02:14:45.825619 30052 maintenance_manager.cc:419] P 155735c7f8b14997ac4c770aee2a9401: Scheduling UndoDeltaBlockGCOp(6635aacafed54994965ad48b2df4f163): 4583 bytes on disk
W20240503 02:14:45.828060 371 consensus_peers.cc:485] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:14:45.828296 29943 maintenance_manager.cc:643] P 155735c7f8b14997ac4c770aee2a9401: UndoDeltaBlockGCOp(6635aacafed54994965ad48b2df4f163) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":1798,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3}
I20240503 02:14:45.830752 30052 maintenance_manager.cc:419] P 155735c7f8b14997ac4c770aee2a9401: Scheduling FlushMRSOp(63b416f4e004490b8e38a96fd065dfea): perf score=0.033408
I20240503 02:14:45.864051 30073 maintenance_manager.cc:643] P a9094240860e43bfb129193f3bfd5be6: FlushMRSOp(1c9200fa969a46e6a48c64293801382b) complete. Timing: real 0.195s user 0.051s sys 0.034s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":7461,"dirs.run_cpu_time_us":219,"dirs.run_wall_time_us":14407,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3706,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":676,"thread_start_us":64506,"threads_started":2,"wal-append.queue_time_us":56357}
I20240503 02:14:45.867031 30182 maintenance_manager.cc:419] P a9094240860e43bfb129193f3bfd5be6: Scheduling UndoDeltaBlockGCOp(1c9200fa969a46e6a48c64293801382b): 4838 bytes on disk
I20240503 02:14:45.867686 30073 maintenance_manager.cc:643] P a9094240860e43bfb129193f3bfd5be6: UndoDeltaBlockGCOp(1c9200fa969a46e6a48c64293801382b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
I20240503 02:14:45.888638 30203 maintenance_manager.cc:643] P 1f10f80f13c84e80b03eda7085a3b012: FlushMRSOp(6635aacafed54994965ad48b2df4f163) complete. Timing: real 0.220s user 0.046s sys 0.006s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":11076,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":2012,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":944,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":22548,"threads_started":2,"wal-append.queue_time_us":11232}
I20240503 02:14:45.891407 30312 maintenance_manager.cc:419] P 1f10f80f13c84e80b03eda7085a3b012: Scheduling FlushMRSOp(1c9200fa969a46e6a48c64293801382b): perf score=0.033430
W20240503 02:14:45.893589 29809 consensus_peers.cc:485] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:14:45.919214 29809 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:14:45.952278 29943 maintenance_manager.cc:643] P 155735c7f8b14997ac4c770aee2a9401: FlushMRSOp(63b416f4e004490b8e38a96fd065dfea) complete. Timing: real 0.121s user 0.064s sys 0.007s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":62,"dirs.run_cpu_time_us":233,"dirs.run_wall_time_us":1562,"drs_written":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4,"lbm_write_time_us":826,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":527,"threads_started":1,"wal-append.queue_time_us":661}
I20240503 02:14:45.954536 30203 maintenance_manager.cc:643] P 1f10f80f13c84e80b03eda7085a3b012: FlushMRSOp(1c9200fa969a46e6a48c64293801382b) complete. Timing: real 0.062s user 0.042s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":37,"dirs.run_cpu_time_us":229,"dirs.run_wall_time_us":1643,"drs_written":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4,"lbm_write_time_us":956,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":3233,"threads_started":1,"wal-append.queue_time_us":4009}
I20240503 02:14:45.955039 30052 maintenance_manager.cc:419] P 155735c7f8b14997ac4c770aee2a9401: Scheduling UndoDeltaBlockGCOp(63b416f4e004490b8e38a96fd065dfea): 4923 bytes on disk
I20240503 02:14:45.955667 29943 maintenance_manager.cc:643] P 155735c7f8b14997ac4c770aee2a9401: UndoDeltaBlockGCOp(63b416f4e004490b8e38a96fd065dfea) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
I20240503 02:14:45.957481 30312 maintenance_manager.cc:419] P 1f10f80f13c84e80b03eda7085a3b012: Scheduling UndoDeltaBlockGCOp(1c9200fa969a46e6a48c64293801382b): 4838 bytes on disk
W20240503 02:14:45.957520 30070 consensus_peers.cc:485] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:14:45.958568 30203 maintenance_manager.cc:643] P 1f10f80f13c84e80b03eda7085a3b012: UndoDeltaBlockGCOp(1c9200fa969a46e6a48c64293801382b) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":243,"lbm_reads_lt_1ms":4}
I20240503 02:14:45.962579 30312 maintenance_manager.cc:419] P 1f10f80f13c84e80b03eda7085a3b012: Scheduling UndoDeltaBlockGCOp(6635aacafed54994965ad48b2df4f163): 4583 bytes on disk
I20240503 02:14:45.963465 30203 maintenance_manager.cc:643] P 1f10f80f13c84e80b03eda7085a3b012: UndoDeltaBlockGCOp(6635aacafed54994965ad48b2df4f163) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
I20240503 02:14:45.991815 30265 consensus_queue.cc:237] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:45.993745 304 raft_consensus.cc:1261] T fa288463ae154e28afd9c71b4b55ef3a P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:14:45.993819 566 raft_consensus.cc:1261] T fa288463ae154e28afd9c71b4b55ef3a P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:14:45.994328 14633 consensus_queue.cc:1035] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:14:45.994828 14266 consensus_queue.cc:1035] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 02:14:46.012637 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:46.012804 15533 raft_consensus.cc:2932] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:46.013430 304 raft_consensus.cc:2932] T fa288463ae154e28afd9c71b4b55ef3a P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:46.021555 29726 catalog_manager.cc:5484] T fa288463ae154e28afd9c71b4b55ef3a P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: config changed from index 6 to 7, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:46.023375 566 raft_consensus.cc:2932] T fa288463ae154e28afd9c71b4b55ef3a P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } }
I20240503 02:14:46.039569 29853 tablet_service.cc:1514] Processing DeleteTablet for tablet fa288463ae154e28afd9c71b4b55ef3a with delete_type TABLET_DATA_TOMBSTONED (TS 16440e365430405fb10366ab70e2b429 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:58744
I20240503 02:14:46.040742 15544 tablet_replica.cc:330] T fa288463ae154e28afd9c71b4b55ef3a P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:14:46.041096 15544 raft_consensus.cc:2229] T fa288463ae154e28afd9c71b4b55ef3a P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:46.041290 15544 raft_consensus.cc:2258] T fa288463ae154e28afd9c71b4b55ef3a P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:46.042305 15544 ts_tablet_manager.cc:1875] T fa288463ae154e28afd9c71b4b55ef3a P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:46.060439 15544 ts_tablet_manager.cc:1888] T fa288463ae154e28afd9c71b4b55ef3a P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:14:46.060577 15544 log.cc:1196] T fa288463ae154e28afd9c71b4b55ef3a P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/fa288463ae154e28afd9c71b4b55ef3a
I20240503 02:14:46.061784 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet fa288463ae154e28afd9c71b4b55ef3a (table rebalance_test_table_9 [id=e80de094521a4d71bc2fb5f3343275b8]) successfully deleted
W20240503 02:14:46.111157 30070 consensus_peers.cc:485] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:14:46.152272 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:46.197368 29809 consensus_peers.cc:485] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:14:46.253479 32708 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.8:42233, user_credentials={real_user=slave}} blocked reactor thread for 54633.1us
W20240503 02:14:46.295099 30070 consensus_peers.cc:485] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:14:46.297842 29940 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:14:46.319509 29809 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:14:46.329620 32709 consensus_peers.cc:485] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:14:46.334390 502 consensus_peers.cc:485] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:14:46.351390 30200 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:14:46.390527 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:46.392652 30244 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:46.398499 29853 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:46.405265 29985 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 02:14:46.426607 30200 consensus_peers.cc:485] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:14:46.453634 30114 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:14:46.484978 547 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:14:46.489954 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240503 02:14:46.543609 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:46.553001 415 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:46.672145 29940 consensus_peers.cc:485] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:14:46.690565 371 consensus_peers.cc:485] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:14:46.705641 29940 consensus_peers.cc:485] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:14:46.716667 29873 consensus_queue.cc:237] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:14:46.718772 304 raft_consensus.cc:1261] T 32a2dbf80daf40d6b21442a39f62f14c P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:14:46.719311 15015 consensus_queue.cc:1035] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:14:46.719360 435 raft_consensus.cc:1261] T 32a2dbf80daf40d6b21442a39f62f14c P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:14:46.719812 15015 consensus_queue.cc:1035] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:14:46.726410 15625 raft_consensus.cc:2932] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:14:46.726927 435 raft_consensus.cc:2932] T 32a2dbf80daf40d6b21442a39f62f14c P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:14:46.726994 304 raft_consensus.cc:2932] T 32a2dbf80daf40d6b21442a39f62f14c P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:14:46.733631 29731 catalog_manager.cc:5484] T 32a2dbf80daf40d6b21442a39f62f14c P 16440e365430405fb10366ab70e2b429 reported cstate change: config changed from index 6 to 7, VOTER a9094240860e43bfb129193f3bfd5be6 (127.22.108.3) evicted. New cstate: current_term: 1 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:14:46.741317 30114 tablet_service.cc:1514] Processing DeleteTablet for tablet 32a2dbf80daf40d6b21442a39f62f14c with delete_type TABLET_DATA_TOMBSTONED (TS a9094240860e43bfb129193f3bfd5be6 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:44812
I20240503 02:14:46.754923 15633 tablet_replica.cc:330] T 32a2dbf80daf40d6b21442a39f62f14c P a9094240860e43bfb129193f3bfd5be6: stopping tablet replica
I20240503 02:14:46.755359 15633 raft_consensus.cc:2229] T 32a2dbf80daf40d6b21442a39f62f14c P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:46.755798 15633 raft_consensus.cc:2258] T 32a2dbf80daf40d6b21442a39f62f14c P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:46.756696 15633 ts_tablet_manager.cc:1875] T 32a2dbf80daf40d6b21442a39f62f14c P a9094240860e43bfb129193f3bfd5be6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:46.764137 15633 ts_tablet_manager.cc:1888] T 32a2dbf80daf40d6b21442a39f62f14c P a9094240860e43bfb129193f3bfd5be6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:14:46.764245 15633 log.cc:1196] T 32a2dbf80daf40d6b21442a39f62f14c P a9094240860e43bfb129193f3bfd5be6: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/wals/32a2dbf80daf40d6b21442a39f62f14c
I20240503 02:14:46.764855 29718 catalog_manager.cc:4830] TS a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): tablet 32a2dbf80daf40d6b21442a39f62f14c (table rebalance_test_table_17 [id=6fdcf94cbd1844948615de6c2a4d18cc]) successfully deleted
W20240503 02:14:46.771169 29809 consensus_peers.cc:485] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:14:46.968261 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:46.993996 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:47.055800 371 consensus_peers.cc:485] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:14:47.083266 29985 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 02:14:47.119588 502 consensus_peers.cc:485] T 36d3f1edbab7477f9b7ac400160491a0 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:14:47.142992 29853 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:47.154465 30070 consensus_peers.cc:485] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:14:47.154986 30244 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:47.170915 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:47.201154 30114 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 02:14:47.255844 29809 consensus_peers.cc:485] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:14:47.272229 415 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:47.308849 502 consensus_peers.cc:485] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:14:47.322458 547 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 02:14:47.329243 30070 consensus_peers.cc:485] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:14:47.352816 371 consensus_peers.cc:485] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:14:47.377074 371 consensus_peers.cc:485] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:14:47.402314 32709 consensus_peers.cc:485] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:14:47.450070 30070 consensus_peers.cc:485] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:14:47.460400 30070 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:14:47.460747 502 consensus_peers.cc:485] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:14:47.519904 30200 consensus_peers.cc:485] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:14:47.535733 30070 consensus_peers.cc:485] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:14:47.546604 29940 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 02:14:47.548204 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:47.560937 29809 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:14:47.591554 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:47.594480 29809 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240503 02:14:47.608464 30200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 56 similar messages]
W20240503 02:14:47.618507 30070 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 02:14:47.620167 30200 consensus_peers.cc:485] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:14:47.652076 502 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 02:14:47.692766 30200 consensus_peers.cc:485] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:14:47.693928 30114 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:14:47.713218 30244 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:47.716771 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240503 02:14:47.721779 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.3:42679, user_credentials={real_user=slave}} blocked reactor thread for 46153.8us
W20240503 02:14:47.727273 502 consensus_peers.cc:485] T ae4373a66af14c8f8a633963dffd2261 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:14:47.729187 29853 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:47.736188 29985 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 02:14:47.750381 30070 consensus_peers.cc:485] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:14:47.757699 547 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 02:14:47.769212 371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 52 similar messages]
I20240503 02:14:47.775565 415 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:47.880002 32709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:14:47.883080 30200 consensus_peers.cc:485] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:14:47.888971 30182 maintenance_manager.cc:419] P a9094240860e43bfb129193f3bfd5be6: Scheduling FlushMRSOp(779a6685fc8c4670a71b51c4493bc2f7): perf score=0.033335
I20240503 02:14:47.922962 30073 maintenance_manager.cc:643] P a9094240860e43bfb129193f3bfd5be6: FlushMRSOp(779a6685fc8c4670a71b51c4493bc2f7) complete. Timing: real 0.034s user 0.028s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":361,"dirs.run_cpu_time_us":222,"dirs.run_wall_time_us":1601,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":823,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":528,"threads_started":2,"wal-append.queue_time_us":341}
I20240503 02:14:47.925915 30182 maintenance_manager.cc:419] P a9094240860e43bfb129193f3bfd5be6: Scheduling UndoDeltaBlockGCOp(779a6685fc8c4670a71b51c4493bc2f7): 4583 bytes on disk
I20240503 02:14:47.926540 30073 maintenance_manager.cc:643] P a9094240860e43bfb129193f3bfd5be6: UndoDeltaBlockGCOp(779a6685fc8c4670a71b51c4493bc2f7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
I20240503 02:14:47.940124 30266 consensus_queue.cc:237] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:14:47.942067 29874 raft_consensus.cc:1261] T 5faccb32e55d4cb689ff491fd20884e6 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:47.942291 565 raft_consensus.cc:1261] T 5faccb32e55d4cb689ff491fd20884e6 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:47.942595 15534 consensus_queue.cc:1035] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:47.943168 15534 consensus_queue.cc:1035] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:47.946177 15534 raft_consensus.cc:2932] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:47.946848 565 raft_consensus.cc:2932] T 5faccb32e55d4cb689ff491fd20884e6 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:47.950788 29873 raft_consensus.cc:2932] T 5faccb32e55d4cb689ff491fd20884e6 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:47.954228 29726 catalog_manager.cc:5484] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 reported cstate change: config changed from index 2 to 3, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) evicted. New cstate: current_term: 1 leader_uuid: "1f10f80f13c84e80b03eda7085a3b012" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:47.964884 29985 tablet_service.cc:1514] Processing DeleteTablet for tablet 5faccb32e55d4cb689ff491fd20884e6 with delete_type TABLET_DATA_TOMBSTONED (TS 155735c7f8b14997ac4c770aee2a9401 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:52158
I20240503 02:14:47.965839 15790 tablet_replica.cc:330] T 5faccb32e55d4cb689ff491fd20884e6 P 155735c7f8b14997ac4c770aee2a9401: stopping tablet replica
I20240503 02:14:47.966074 15790 raft_consensus.cc:2229] T 5faccb32e55d4cb689ff491fd20884e6 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:47.966320 15790 raft_consensus.cc:2258] T 5faccb32e55d4cb689ff491fd20884e6 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:47.966996 15790 ts_tablet_manager.cc:1875] T 5faccb32e55d4cb689ff491fd20884e6 P 155735c7f8b14997ac4c770aee2a9401: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:47.972681 15790 ts_tablet_manager.cc:1888] T 5faccb32e55d4cb689ff491fd20884e6 P 155735c7f8b14997ac4c770aee2a9401: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:14:47.972900 15790 log.cc:1196] T 5faccb32e55d4cb689ff491fd20884e6 P 155735c7f8b14997ac4c770aee2a9401: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/wals/5faccb32e55d4cb689ff491fd20884e6
I20240503 02:14:47.974056 29719 catalog_manager.cc:4830] TS 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): tablet 5faccb32e55d4cb689ff491fd20884e6 (table rebalancer_extra_table_50 [id=e4dd29f93ff64f6088ff5503d1f570dd]) successfully deleted
I20240503 02:14:47.978307 30052 maintenance_manager.cc:419] P 155735c7f8b14997ac4c770aee2a9401: Scheduling FlushMRSOp(3f00949b4ca84a49b0f6e62bd7373586): perf score=0.033342
I20240503 02:14:47.988569 30312 maintenance_manager.cc:419] P 1f10f80f13c84e80b03eda7085a3b012: Scheduling FlushMRSOp(779a6685fc8c4670a71b51c4493bc2f7): perf score=0.033344
W20240503 02:14:47.998791 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:48.014714 32709 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:14:48.035053 29921 maintenance_manager.cc:419] P 16440e365430405fb10366ab70e2b429: Scheduling FlushMRSOp(3f00949b4ca84a49b0f6e62bd7373586): perf score=0.033350
W20240503 02:14:48.044811 502 consensus_peers.cc:485] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:14:48.055655 29940 consensus_peers.cc:485] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:14:48.062387 29943 maintenance_manager.cc:643] P 155735c7f8b14997ac4c770aee2a9401: FlushMRSOp(3f00949b4ca84a49b0f6e62bd7373586) complete. Timing: real 0.084s user 0.053s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":7763,"dirs.run_cpu_time_us":181,"dirs.run_wall_time_us":8895,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1277,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":14110,"threads_started":2,"wal-append.queue_time_us":6368}
I20240503 02:14:48.065943 30052 maintenance_manager.cc:419] P 155735c7f8b14997ac4c770aee2a9401: Scheduling UndoDeltaBlockGCOp(3f00949b4ca84a49b0f6e62bd7373586): 4836 bytes on disk
I20240503 02:14:48.066624 29943 maintenance_manager.cc:643] P 155735c7f8b14997ac4c770aee2a9401: UndoDeltaBlockGCOp(3f00949b4ca84a49b0f6e62bd7373586) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
W20240503 02:14:48.079231 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:48.096351 30203 maintenance_manager.cc:643] P 1f10f80f13c84e80b03eda7085a3b012: FlushMRSOp(779a6685fc8c4670a71b51c4493bc2f7) complete. Timing: real 0.107s user 0.040s sys 0.002s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":331,"dirs.run_cpu_time_us":190,"dirs.run_wall_time_us":4755,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":735,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":21914,"threads_started":2,"wal-append.queue_time_us":21773}
I20240503 02:14:48.102428 30312 maintenance_manager.cc:419] P 1f10f80f13c84e80b03eda7085a3b012: Scheduling UndoDeltaBlockGCOp(779a6685fc8c4670a71b51c4493bc2f7): 4583 bytes on disk
I20240503 02:14:48.104360 30203 maintenance_manager.cc:643] P 1f10f80f13c84e80b03eda7085a3b012: UndoDeltaBlockGCOp(779a6685fc8c4670a71b51c4493bc2f7) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":545,"lbm_reads_lt_1ms":4}
I20240503 02:14:48.113337 29812 maintenance_manager.cc:643] P 16440e365430405fb10366ab70e2b429: FlushMRSOp(3f00949b4ca84a49b0f6e62bd7373586) complete. Timing: real 0.078s user 0.040s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":339,"dirs.run_cpu_time_us":172,"dirs.run_wall_time_us":1541,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":728,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":16898,"threads_started":2,"wal-append.queue_time_us":16749}
I20240503 02:14:48.115757 29921 maintenance_manager.cc:419] P 16440e365430405fb10366ab70e2b429: Scheduling UndoDeltaBlockGCOp(3f00949b4ca84a49b0f6e62bd7373586): 4836 bytes on disk
I20240503 02:14:48.116392 29812 maintenance_manager.cc:643] P 16440e365430405fb10366ab70e2b429: UndoDeltaBlockGCOp(3f00949b4ca84a49b0f6e62bd7373586) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
I20240503 02:14:48.118690 29921 maintenance_manager.cc:419] P 16440e365430405fb10366ab70e2b429: Scheduling FlushMRSOp(32a2dbf80daf40d6b21442a39f62f14c): perf score=0.033371
I20240503 02:14:48.161110 29812 maintenance_manager.cc:643] P 16440e365430405fb10366ab70e2b429: FlushMRSOp(32a2dbf80daf40d6b21442a39f62f14c) complete. Timing: real 0.042s user 0.041s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":58,"dirs.run_cpu_time_us":313,"dirs.run_wall_time_us":1504,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":709,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":3549,"threads_started":1,"wal-append.queue_time_us":3671}
I20240503 02:14:48.163867 29921 maintenance_manager.cc:419] P 16440e365430405fb10366ab70e2b429: Scheduling UndoDeltaBlockGCOp(32a2dbf80daf40d6b21442a39f62f14c): 4921 bytes on disk
I20240503 02:14:48.164500 29812 maintenance_manager.cc:643] P 16440e365430405fb10366ab70e2b429: UndoDeltaBlockGCOp(32a2dbf80daf40d6b21442a39f62f14c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
W20240503 02:14:48.187711 29809 consensus_peers.cc:485] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:14:48.396343 29940 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:14:48.426553 29809 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:14:48.426822 29985 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:48.464381 29853 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:48.471563 502 consensus_peers.cc:485] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:14:48.483960 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:48.502295 30244 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:48.540398 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:48.541594 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:48.562767 30114 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:14:48.595825 547 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 02:14:48.602320 371 consensus_peers.cc:485] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:14:48.610262 415 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:48.687515 29809 consensus_peers.cc:485] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:14:48.692603 30070 consensus_peers.cc:485] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:14:48.700666 30070 consensus_peers.cc:485] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:14:48.716699 30070 consensus_peers.cc:485] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:14:48.768534 29809 consensus_peers.cc:485] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:14:48.783586 30200 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:14:48.820925 29809 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:14:48.866236 32709 consensus_peers.cc:485] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:14:48.927265 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:48.943477 29853 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:48.976898 29985 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:48.979575 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:48.987674 30114 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 02:14:48.989085 371 consensus_peers.cc:485] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:14:48.989993 547 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:14:49.021675 30244 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:49.053484 415 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:49.079138 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:49.163028 30200 consensus_peers.cc:485] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:14:49.219830 29940 consensus_peers.cc:485] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:14:49.408020 29809 consensus_peers.cc:485] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:14:49.413708 29985 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:49.413900 29853 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:49.415006 30244 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:49.462396 502 consensus_peers.cc:485] T 36d3f1edbab7477f9b7ac400160491a0 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:14:49.465245 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:49.466331 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240503 02:14:49.516834 29940 consensus_peers.cc:485] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:14:49.526304 415 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:49.536226 547 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:14:49.561455 30114 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 02:14:49.620126 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:49.687232 502 consensus_peers.cc:485] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:14:49.710110 32709 consensus_peers.cc:485] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:14:49.724615 30070 consensus_peers.cc:485] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:14:49.754669 29809 consensus_peers.cc:485] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:14:49.810416 371 consensus_peers.cc:485] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:14:49.913655 30070 consensus_peers.cc:485] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:14:49.915812 30200 consensus_peers.cc:485] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:14:49.934513 30070 consensus_peers.cc:485] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:14:49.939842 32709 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.4:33935, user_credentials={real_user=slave}} blocked reactor thread for 54236.8us
W20240503 02:14:49.945200 371 consensus_peers.cc:485] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:14:49.950079 32708 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.8:42233, user_credentials={real_user=slave}} blocked reactor thread for 54580.9us
W20240503 02:14:49.950176 502 consensus_peers.cc:485] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:14:49.958811 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:50.005759 29853 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:50.028577 371 consensus_peers.cc:485] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:14:50.031324 29985 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 02:14:50.031492 30070 consensus_peers.cc:485] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:14:50.032100 30244 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:50.042773 30114 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:14:50.085115 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:50.091750 547 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 02:14:50.113343 30200 consensus_peers.cc:485] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:14:50.123787 30200 consensus_peers.cc:485] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:14:50.127439 415 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:50.137861 30070 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:14:50.203434 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:50.209740 30070 consensus_peers.cc:485] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:14:50.211532 32709 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:14:50.235198 29809 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:14:50.276335 435 consensus_queue.cc:237] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:14:50.278383 30265 raft_consensus.cc:1261] T 52a541b1d3db4228bc5a27af521f0b93 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:50.278400 567 raft_consensus.cc:1261] T 52a541b1d3db4228bc5a27af521f0b93 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:14:50.278952 14880 consensus_queue.cc:1035] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:50.279815 15671 consensus_queue.cc:1035] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:14:50.282778 14880 raft_consensus.cc:2932] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:14:50.285861 567 raft_consensus.cc:2932] T 52a541b1d3db4228bc5a27af521f0b93 P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:14:50.286371 30265 raft_consensus.cc:2932] T 52a541b1d3db4228bc5a27af521f0b93 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:14:50.288630 29733 catalog_manager.cc:5484] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: config changed from index 2 to 3, VOTER 16440e365430405fb10366ab70e2b429 (127.22.108.1) evicted. New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
W20240503 02:14:50.294274 29940 consensus_peers.cc:485] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:14:50.296308 30200 consensus_peers.cc:485] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:14:50.301091 29853 tablet_service.cc:1514] Processing DeleteTablet for tablet 52a541b1d3db4228bc5a27af521f0b93 with delete_type TABLET_DATA_TOMBSTONED (TS 16440e365430405fb10366ab70e2b429 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:58744
I20240503 02:14:50.309456 16071 tablet_replica.cc:330] T 52a541b1d3db4228bc5a27af521f0b93 P 16440e365430405fb10366ab70e2b429: stopping tablet replica
I20240503 02:14:50.309705 16071 raft_consensus.cc:2229] T 52a541b1d3db4228bc5a27af521f0b93 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:50.309955 16071 raft_consensus.cc:2258] T 52a541b1d3db4228bc5a27af521f0b93 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:50.310669 16071 ts_tablet_manager.cc:1875] T 52a541b1d3db4228bc5a27af521f0b93 P 16440e365430405fb10366ab70e2b429: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:50.316644 16071 ts_tablet_manager.cc:1888] T 52a541b1d3db4228bc5a27af521f0b93 P 16440e365430405fb10366ab70e2b429: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:14:50.316797 16071 log.cc:1196] T 52a541b1d3db4228bc5a27af521f0b93 P 16440e365430405fb10366ab70e2b429: Deleting WAL directory at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/wals/52a541b1d3db4228bc5a27af521f0b93
I20240503 02:14:50.317714 29721 catalog_manager.cc:4830] TS 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): tablet 52a541b1d3db4228bc5a27af521f0b93 (table rebalancer_extra_table_52 [id=7feb3c0f2dd74037bb83d98b5b0b5520]) successfully deleted
W20240503 02:14:50.332466 502 consensus_peers.cc:485] T ae4373a66af14c8f8a633963dffd2261 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:14:50.390779 502 consensus_peers.cc:485] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:14:50.513532 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:50.578259 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:50.579150 29809 consensus_peers.cc:485] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:14:50.755622 547 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:14:50.762636 29853 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:50.766808 30114 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:14:50.768162 29985 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:50.771966 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:50.794132 415 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:50.797775 30244 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:50.871757 502 consensus_peers.cc:485] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:14:50.938997 30070 consensus_peers.cc:485] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:14:50.999261 29809 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:14:51.012450 29940 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:14:51.013080 29809 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:14:51.027051 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:51.077512 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:51.106375 371 consensus_peers.cc:485] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:14:51.169906 29809 consensus_peers.cc:485] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:14:51.215560 29985 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:51.279024 29853 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:51.316613 30070 consensus_peers.cc:485] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:14:51.329653 30244 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:51.350392 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:51.358877 30052 maintenance_manager.cc:419] P 155735c7f8b14997ac4c770aee2a9401: Scheduling FlushMRSOp(08ee37b160d441ddabb640be907d6771): perf score=0.033345
W20240503 02:14:51.359164 29809 consensus_peers.cc:485] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:14:51.392879 415 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:51.400748 30312 maintenance_manager.cc:419] P 1f10f80f13c84e80b03eda7085a3b012: Scheduling FlushMRSOp(08ee37b160d441ddabb640be907d6771): perf score=0.033343
I20240503 02:14:51.402061 30114 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:14:51.411309 547 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:14:51.454489 29921 maintenance_manager.cc:419] P 16440e365430405fb10366ab70e2b429: Scheduling FlushMRSOp(5040203bd48549ff9c8f64c71d6bc598): perf score=0.033357
W20240503 02:14:51.455713 30200 consensus_peers.cc:485] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:14:51.463701 30182 maintenance_manager.cc:419] P a9094240860e43bfb129193f3bfd5be6: Scheduling FlushMRSOp(5040203bd48549ff9c8f64c71d6bc598): perf score=0.033362
I20240503 02:14:51.495431 30203 maintenance_manager.cc:643] P 1f10f80f13c84e80b03eda7085a3b012: FlushMRSOp(08ee37b160d441ddabb640be907d6771) complete. Timing: real 0.094s user 0.045s sys 0.031s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":7339,"dirs.run_cpu_time_us":219,"dirs.run_wall_time_us":6391,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":849,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":40737,"threads_started":2,"wal-append.queue_time_us":33139}
I20240503 02:14:51.498288 30312 maintenance_manager.cc:419] P 1f10f80f13c84e80b03eda7085a3b012: Scheduling UndoDeltaBlockGCOp(08ee37b160d441ddabb640be907d6771): 4583 bytes on disk
I20240503 02:14:51.499068 30203 maintenance_manager.cc:643] P 1f10f80f13c84e80b03eda7085a3b012: UndoDeltaBlockGCOp(08ee37b160d441ddabb640be907d6771) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
W20240503 02:14:51.505336 32709 consensus_peers.cc:485] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:14:51.509882 30070 consensus_peers.cc:485] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:14:51.517519 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:51.525925 29943 maintenance_manager.cc:643] P 155735c7f8b14997ac4c770aee2a9401: FlushMRSOp(08ee37b160d441ddabb640be907d6771) complete. Timing: real 0.167s user 0.051s sys 0.015s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":1574,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":1375,"drs_written":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1100,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":8416,"threads_started":2,"wal-append.queue_time_us":861}
I20240503 02:14:51.528656 30052 maintenance_manager.cc:419] P 155735c7f8b14997ac4c770aee2a9401: Scheduling FlushMRSOp(c5cd64614c53487ea1c6cc4a4ca5c379): perf score=0.033382
W20240503 02:14:51.572552 30200 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:14:51.580812 29943 maintenance_manager.cc:643] P 155735c7f8b14997ac4c770aee2a9401: FlushMRSOp(c5cd64614c53487ea1c6cc4a4ca5c379) complete. Timing: real 0.052s user 0.032s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":1333,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":705,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":11297,"threads_started":1,"wal-append.queue_time_us":11260}
I20240503 02:14:51.583601 30052 maintenance_manager.cc:419] P 155735c7f8b14997ac4c770aee2a9401: Scheduling UndoDeltaBlockGCOp(c5cd64614c53487ea1c6cc4a4ca5c379): 4923 bytes on disk
I20240503 02:14:51.584338 29943 maintenance_manager.cc:643] P 155735c7f8b14997ac4c770aee2a9401: UndoDeltaBlockGCOp(c5cd64614c53487ea1c6cc4a4ca5c379) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":128,"lbm_reads_lt_1ms":4}
I20240503 02:14:51.584236 30073 maintenance_manager.cc:643] P a9094240860e43bfb129193f3bfd5be6: FlushMRSOp(5040203bd48549ff9c8f64c71d6bc598) complete. Timing: real 0.120s user 0.050s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":313,"dirs.run_cpu_time_us":249,"dirs.run_wall_time_us":1496,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":830,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":17418,"threads_started":2,"wal-append.queue_time_us":17355}
I20240503 02:14:51.587042 30182 maintenance_manager.cc:419] P a9094240860e43bfb129193f3bfd5be6: Scheduling FlushMRSOp(c5cd64614c53487ea1c6cc4a4ca5c379): perf score=0.033391
I20240503 02:14:51.589526 30052 maintenance_manager.cc:419] P 155735c7f8b14997ac4c770aee2a9401: Scheduling UndoDeltaBlockGCOp(08ee37b160d441ddabb640be907d6771): 4583 bytes on disk
I20240503 02:14:51.590327 29943 maintenance_manager.cc:643] P 155735c7f8b14997ac4c770aee2a9401: UndoDeltaBlockGCOp(08ee37b160d441ddabb640be907d6771) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
I20240503 02:14:51.593895 29812 maintenance_manager.cc:643] P 16440e365430405fb10366ab70e2b429: FlushMRSOp(5040203bd48549ff9c8f64c71d6bc598) complete. Timing: real 0.139s user 0.044s sys 0.019s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":6387,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":11088,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":966,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":17506,"threads_started":2,"wal-append.queue_time_us":7079}
I20240503 02:14:51.596612 29921 maintenance_manager.cc:419] P 16440e365430405fb10366ab70e2b429: Scheduling UndoDeltaBlockGCOp(5040203bd48549ff9c8f64c71d6bc598): 4836 bytes on disk
I20240503 02:14:51.597385 29812 maintenance_manager.cc:643] P 16440e365430405fb10366ab70e2b429: UndoDeltaBlockGCOp(5040203bd48549ff9c8f64c71d6bc598) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
W20240503 02:14:51.607393 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:51.650950 30073 maintenance_manager.cc:643] P a9094240860e43bfb129193f3bfd5be6: FlushMRSOp(c5cd64614c53487ea1c6cc4a4ca5c379) complete. Timing: real 0.063s user 0.046s sys 0.011s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":206,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":1574,"drs_written":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1609,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":2126,"threads_started":1,"wal-append.queue_time_us":2023}
I20240503 02:14:51.653757 30182 maintenance_manager.cc:419] P a9094240860e43bfb129193f3bfd5be6: Scheduling UndoDeltaBlockGCOp(c5cd64614c53487ea1c6cc4a4ca5c379): 4923 bytes on disk
I20240503 02:14:51.654718 30073 maintenance_manager.cc:643] P a9094240860e43bfb129193f3bfd5be6: UndoDeltaBlockGCOp(c5cd64614c53487ea1c6cc4a4ca5c379) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240503 02:14:51.657127 30182 maintenance_manager.cc:419] P a9094240860e43bfb129193f3bfd5be6: Scheduling UndoDeltaBlockGCOp(5040203bd48549ff9c8f64c71d6bc598): 4836 bytes on disk
I20240503 02:14:51.657672 30073 maintenance_manager.cc:643] P a9094240860e43bfb129193f3bfd5be6: UndoDeltaBlockGCOp(5040203bd48549ff9c8f64c71d6bc598) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
W20240503 02:14:51.663872 371 consensus_peers.cc:485] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:14:51.829090 502 consensus_peers.cc:485] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:14:51.845849 29809 consensus_peers.cc:485] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:14:51.908572 29940 consensus_peers.cc:485] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:14:51.927376 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:51.995903 29853 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:51.998207 30244 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:51.999565 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:52.020948 30114 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:14:52.028362 29985 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 02:14:52.031471 502 consensus_peers.cc:485] T 36d3f1edbab7477f9b7ac400160491a0 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:14:52.058719 547 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:14:52.124665 415 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:52.154680 29809 consensus_peers.cc:485] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:14:52.172526 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:52.242550 371 consensus_peers.cc:485] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:14:52.282668 29940 consensus_peers.cc:485] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:14:52.283366 30070 consensus_peers.cc:485] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:14:52.299284 30070 consensus_peers.cc:485] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:14:52.330099 30200 consensus_peers.cc:485] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:14:52.349310 502 consensus_peers.cc:485] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:14:52.363119 371 consensus_peers.cc:485] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:14:52.421058 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:52.498880 30200 consensus_peers.cc:485] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:14:52.547863 29985 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 02:14:52.556322 371 consensus_peers.cc:485] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:14:52.559473 30070 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:14:52.566531 30200 consensus_peers.cc:485] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:14:52.571064 29853 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:52.579182 30070 consensus_peers.cc:485] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:14:52.579291 30070 consensus_peers.cc:485] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:14:52.597296 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:52.599545 30244 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:52.600549 29940 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 02:14:52.601482 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:52.619889 32709 consensus_peers.cc:485] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:14:52.629966 30200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 02:14:52.643971 29809 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240503 02:14:52.660632 30070 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240503 02:14:52.661729 30070 consensus_peers.cc:485] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:14:52.663033 30114 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:14:52.682334 415 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:52.723623 30200 consensus_peers.cc:485] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:14:52.765619 547 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 02:14:52.781168 502 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 02:14:52.788722 502 consensus_peers.cc:485] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:14:52.802367 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:52.834298 371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:14:52.860599 502 consensus_peers.cc:485] T ae4373a66af14c8f8a633963dffd2261 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:14:52.865773 29809 consensus_peers.cc:485] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:14:52.885512 32709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:14:52.886343 32709 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:14:52.998885 29809 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:14:53.063632 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:53.065459 30244 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:53.067014 30114 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:14:53.094799 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:53.096640 29985 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 02:14:53.116093 29940 consensus_peers.cc:485] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:14:53.138708 415 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:53.140187 29853 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:53.148625 547 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 02:14:53.389063 29940 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:14:53.426259 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:53.426765 502 consensus_peers.cc:485] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:14:53.460908 30070 consensus_peers.cc:485] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:14:53.515511 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:53.592768 29853 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:53.614462 30244 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:53.619510 415 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:53.624253 29985 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:53.633786 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:53.637423 30114 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:14:53.649420 547 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 02:14:53.653764 29809 consensus_peers.cc:485] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:14:53.653906 29809 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:14:53.669737 371 consensus_peers.cc:485] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:14:53.706732 29809 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:14:53.791031 29809 consensus_peers.cc:485] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:14:53.889173 32709 consensus_peers.cc:485] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:14:53.980679 30070 consensus_peers.cc:485] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:14:53.992868 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:54.000715 30200 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:14:54.015323 30200 consensus_peers.cc:485] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:14:54.045360 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:54.048452 29853 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:54.076833 30244 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:54.099779 30114 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:14:54.112147 547 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:14:54.112859 29985 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:54.120357 415 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:54.125404 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240503 02:14:54.127996 29809 consensus_peers.cc:485] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:14:54.134220 371 consensus_peers.cc:485] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:14:54.215607 30070 consensus_peers.cc:485] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:14:54.423910 29809 consensus_peers.cc:485] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:14:54.433290 29940 consensus_peers.cc:485] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:14:54.448180 502 consensus_peers.cc:485] T 36d3f1edbab7477f9b7ac400160491a0 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:14:54.505411 502 consensus_peers.cc:485] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:14:54.570070 32708 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.8:42233, user_credentials={real_user=slave}} blocked reactor thread for 50744.3us
W20240503 02:14:54.577626 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:54.590795 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:54.600324 371 consensus_peers.cc:485] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:14:54.647305 502 consensus_peers.cc:485] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:14:54.648038 29985 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:54.665422 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:54.668530 29853 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:54.669417 30114 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:14:54.697764 547 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:14:54.719350 415 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:54.741138 30244 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:54.771739 371 consensus_peers.cc:485] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:14:54.863610 30200 consensus_peers.cc:485] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:14:54.890358 30070 consensus_peers.cc:485] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:14:54.906952 30070 consensus_peers.cc:485] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:14:54.911535 29940 consensus_peers.cc:485] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:14:55.014613 30070 consensus_peers.cc:485] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:14:55.055682 30070 consensus_peers.cc:485] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:14:55.057965 30200 consensus_peers.cc:485] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:14:55.075768 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:55.080706 502 consensus_peers.cc:485] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:14:55.082409 30070 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:14:55.091434 30070 consensus_peers.cc:485] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:14:55.102161 371 consensus_peers.cc:485] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:14:55.140376 29985 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 02:14:55.143594 30200 consensus_peers.cc:485] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:14:55.150043 32709 consensus_peers.cc:485] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:14:55.157851 30200 consensus_peers.cc:485] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:14:55.171207 29853 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:55.175248 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240503 02:14:55.185551 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:55.210588 547 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:14:55.222189 30244 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:55.352097 415 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:55.389372 30114 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 02:14:55.434228 502 consensus_peers.cc:485] T ae4373a66af14c8f8a633963dffd2261 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:14:55.463232 32709 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:14:55.464183 29809 consensus_peers.cc:485] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:14:55.488271 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:55.659324 29809 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:14:55.701861 29940 consensus_peers.cc:485] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:14:55.717896 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:55.772783 29853 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:55.796247 30114 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 02:14:55.812201 29940 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:14:55.814410 547 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:14:55.814410 30244 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:55.814865 29985 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:55.832046 415 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:55.849628 502 consensus_peers.cc:485] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:14:55.886385 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240503 02:14:55.970448 30070 consensus_peers.cc:485] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:14:56.034703 371 consensus_peers.cc:485] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:14:56.037524 29809 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:14:56.085260 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:56.108106 29809 consensus_peers.cc:485] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:14:56.166545 29809 consensus_peers.cc:485] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:14:56.271828 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:56.294533 29853 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:56.309897 29985 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:56.321457 30244 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:56.331573 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:56.376525 30114 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:14:56.384243 415 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:56.414640 547 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 02:14:56.455987 32709 consensus_peers.cc:485] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:14:56.478981 29809 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:14:56.513710 29809 consensus_peers.cc:485] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:14:56.513738 30070 consensus_peers.cc:485] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:14:56.515585 30070 consensus_peers.cc:485] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:14:56.528719 30200 consensus_peers.cc:485] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:14:56.591528 30200 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:14:56.603480 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:56.609464 371 consensus_peers.cc:485] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:14:56.835474 29853 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:56.848063 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:56.850692 30244 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:56.864908 29985 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:14:56.872776 30114 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:14:56.883234 547 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:14:56.901791 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:56.903558 415 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:56.942337 502 consensus_peers.cc:485] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:14:56.950174 29809 consensus_peers.cc:485] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:14:57.005802 371 consensus_peers.cc:485] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:14:57.064591 502 consensus_peers.cc:485] T 36d3f1edbab7477f9b7ac400160491a0 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:14:57.095605 502 consensus_peers.cc:485] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:14:57.181903 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:57.222357 29940 consensus_peers.cc:485] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:14:57.237970 371 consensus_peers.cc:485] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:14:57.290652 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:57.313241 30070 consensus_peers.cc:485] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:14:57.357409 29853 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:57.362691 30070 consensus_peers.cc:485] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:14:57.391674 29985 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 02:14:57.395081 29940 consensus_peers.cc:485] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:14:57.407974 30114 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:14:57.414494 30244 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:14:57.419010 547 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:14:57.445566 32755 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:14:57.451295 415 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:14:57.473593 30200 consensus_peers.cc:485] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:14:57.526165 371 consensus_peers.cc:485] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:14:57.528206 502 consensus_peers.cc:485] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:14:57.548350 30070 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:14:57.566027 30200 consensus_peers.cc:485] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:14:57.570183 30070 consensus_peers.cc:485] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:14:57.580657 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:57.593900 30070 consensus_peers.cc:485] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:14:57.630568 30070 consensus_peers.cc:485] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:14:57.654640 30200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 02:14:57.655794 30200 consensus_peers.cc:485] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:14:57.663635 30134 consensus_queue.cc:237] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
W20240503 02:14:57.666523 30070 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 90 similar messages]
I20240503 02:14:57.666759 30004 raft_consensus.cc:1261] T c5cd64614c53487ea1c6cc4a4ca5c379 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240503 02:14:57.666869 30070 consensus_peers.cc:485] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 -> Peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241): Couldn't send request to peer fbbc2be6382e47908ecb7c68c367bd26. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c5cd64614c53487ea1c6cc4a4ca5c379. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:57.667237 16047 consensus_queue.cc:1035] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 02:14:57.668440 30070 consensus_peers.cc:485] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:14:57.678992 29940 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 57 similar messages]
W20240503 02:14:57.682806 29809 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240503 02:14:57.683862 29809 consensus_peers.cc:485] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:14:57.713943 32709 consensus_peers.cc:485] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:14:57.778899 30200 consensus_peers.cc:485] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:14:57.818272 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:57.912261 502 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 02:14:57.914566 371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:14:57.971282 32709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240503 02:14:58.017261 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:58.058480 32709 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:14:58.127521 16930 ts_tablet_manager.cc:902] T c5cd64614c53487ea1c6cc4a4ca5c379 P fbbc2be6382e47908ecb7c68c367bd26: Initiating tablet copy from peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:14:58.128988 16930 tablet_copy_client.cc:308] T c5cd64614c53487ea1c6cc4a4ca5c379 P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.3:42679
I20240503 02:14:58.129616 30156 tablet_copy_service.cc:140] P a9094240860e43bfb129193f3bfd5be6: Received BeginTabletCopySession request for tablet c5cd64614c53487ea1c6cc4a4ca5c379 from peer fbbc2be6382e47908ecb7c68c367bd26 ({username='slave'} at 127.22.108.7:37671)
I20240503 02:14:58.129783 30156 tablet_copy_service.cc:161] P a9094240860e43bfb129193f3bfd5be6: Beginning new tablet copy session on tablet c5cd64614c53487ea1c6cc4a4ca5c379 from peer fbbc2be6382e47908ecb7c68c367bd26 at {username='slave'} at 127.22.108.7:37671: session id = fbbc2be6382e47908ecb7c68c367bd26-c5cd64614c53487ea1c6cc4a4ca5c379
I20240503 02:14:58.132004 30156 tablet_copy_source_session.cc:205] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:14:58.133215 16930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5cd64614c53487ea1c6cc4a4ca5c379. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:58.138612 16930 tablet_copy_client.cc:783] T c5cd64614c53487ea1c6cc4a4ca5c379 P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Starting download of 5 data blocks...
I20240503 02:14:58.144194 16930 tablet_copy_client.cc:647] T c5cd64614c53487ea1c6cc4a4ca5c379 P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:58.145992 16930 tablet_copy_client.cc:523] T c5cd64614c53487ea1c6cc4a4ca5c379 P fbbc2be6382e47908ecb7c68c367bd26: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:58.149217 16930 tablet_bootstrap.cc:492] T c5cd64614c53487ea1c6cc4a4ca5c379 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap starting.
I20240503 02:14:58.164364 16930 tablet_bootstrap.cc:492] T c5cd64614c53487ea1c6cc4a4ca5c379 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=5 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 02:14:58.164835 16930 tablet_bootstrap.cc:492] T c5cd64614c53487ea1c6cc4a4ca5c379 P fbbc2be6382e47908ecb7c68c367bd26: Bootstrap complete.
I20240503 02:14:58.165112 16930 ts_tablet_manager.cc:1367] T c5cd64614c53487ea1c6cc4a4ca5c379 P fbbc2be6382e47908ecb7c68c367bd26: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.003s
I20240503 02:14:58.165800 16930 raft_consensus.cc:348] T c5cd64614c53487ea1c6cc4a4ca5c379 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:14:58.166245 16930 raft_consensus.cc:729] T c5cd64614c53487ea1c6cc4a4ca5c379 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Initialized, Role: FOLLOWER
I20240503 02:14:58.166416 16930 consensus_queue.cc:260] T c5cd64614c53487ea1c6cc4a4ca5c379 P fbbc2be6382e47908ecb7c68c367bd26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:14:58.168349 16930 ts_tablet_manager.cc:1398] T c5cd64614c53487ea1c6cc4a4ca5c379 P fbbc2be6382e47908ecb7c68c367bd26: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:14:58.169066 30156 tablet_copy_service.cc:342] P a9094240860e43bfb129193f3bfd5be6: Request end of tablet copy session fbbc2be6382e47908ecb7c68c367bd26-c5cd64614c53487ea1c6cc4a4ca5c379 received from {username='slave'} at 127.22.108.7:37671
I20240503 02:14:58.169193 30156 tablet_copy_service.cc:433] P a9094240860e43bfb129193f3bfd5be6: ending tablet copy session fbbc2be6382e47908ecb7c68c367bd26-c5cd64614c53487ea1c6cc4a4ca5c379 on tablet c5cd64614c53487ea1c6cc4a4ca5c379 with peer fbbc2be6382e47908ecb7c68c367bd26
W20240503 02:14:58.184065 502 consensus_peers.cc:485] T ae4373a66af14c8f8a633963dffd2261 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:14:58.212078 29809 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:14:58.262040 29940 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:14:58.286669 483 maintenance_manager.cc:419] P fbbc2be6382e47908ecb7c68c367bd26: Scheduling UndoDeltaBlockGCOp(c5cd64614c53487ea1c6cc4a4ca5c379): 4923 bytes on disk
I20240503 02:14:58.287293 374 maintenance_manager.cc:643] P fbbc2be6382e47908ecb7c68c367bd26: UndoDeltaBlockGCOp(c5cd64614c53487ea1c6cc4a4ca5c379) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
W20240503 02:14:58.357841 29940 consensus_peers.cc:485] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:14:58.402573 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:58.440591 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:58.495152 30070 consensus_peers.cc:485] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:14:58.503243 29809 consensus_peers.cc:485] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:14:58.504689 502 consensus_peers.cc:485] T eec091fa0ba947d68577d562e538ab56 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:14:58.530928 29809 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:14:58.583534 29809 consensus_peers.cc:485] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:14:58.708094 435 raft_consensus.cc:1203] T c5cd64614c53487ea1c6cc4a4ca5c379 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 02:14:58.709201 16942 raft_consensus.cc:2932] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:14:58.709820 435 raft_consensus.cc:2932] T c5cd64614c53487ea1c6cc4a4ca5c379 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:14:58.713374 16942 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:42036 (request call id 0) took 1050 ms. Trace:
I20240503 02:14:58.713588 16942 rpcz_store.cc:276] 0503 02:14:57.662980 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 02:14:57.663045 (+ 65us) service_pool.cc:225] Handling call
0503 02:14:57.664630 (+ 1585us) log.cc:842] Serialized 482 byte log entry
0503 02:14:58.713322 (+1048692us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":110,"mutex_wait_us":76,"raft.queue_time_us":457,"raft.run_cpu_time_us":482,"raft.run_wall_time_us":481,"spinlock_wait_cycles":60160,"thread_start_us":8336,"threads_started":2,"wal-append.queue_time_us":327,"wal-append.run_cpu_time_us":502,"wal-append.run_wall_time_us":1002724}
I20240503 02:14:58.718420 29725 catalog_manager.cc:5484] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 reported cstate change: config changed from index 5 to 6, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New cstate: current_term: 1 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
W20240503 02:14:58.720206 371 consensus_peers.cc:485] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:14:58.724368 30005 raft_consensus.cc:2932] T c5cd64614c53487ea1c6cc4a4ca5c379 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
W20240503 02:14:58.736554 29809 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:14:58.789458 435 consensus_queue.cc:237] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:14:58.792529 307 raft_consensus.cc:1261] T 2baf70e0e1894cea9fdeb92279bc0cff P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:14:58.793097 14880 consensus_queue.cc:1035] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:14:58.794673 370 consensus_peers.cc:485] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2baf70e0e1894cea9fdeb92279bc0cff. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:58.794661 30135 raft_consensus.cc:1261] T 2baf70e0e1894cea9fdeb92279bc0cff P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:14:58.795388 14880 consensus_queue.cc:1035] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:14:58.797962 14880 raft_consensus.cc:2932] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:58.799091 30135 raft_consensus.cc:2932] T 2baf70e0e1894cea9fdeb92279bc0cff P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:58.803325 307 raft_consensus.cc:2932] T 2baf70e0e1894cea9fdeb92279bc0cff P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:14:58.804157 29726 catalog_manager.cc:5484] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: config changed from index 3 to 4, VOTER f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) added. New cstate: current_term: 1 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
W20240503 02:14:58.821612 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:58.849758 30135 consensus_queue.cc:237] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
W20240503 02:14:58.858342 30070 consensus_peers.cc:485] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 -> Peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): Couldn't send request to peer 1f10f80f13c84e80b03eda7085a3b012. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b3b9d83e4c742bfa62624676cbb6a11. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:58.858589 307 raft_consensus.cc:1261] T 9b3b9d83e4c742bfa62624676cbb6a11 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20240503 02:14:58.859206 30070 consensus_peers.cc:485] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:58.859550 16944 consensus_queue.cc:1035] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 02:14:58.909107 32709 consensus_peers.cc:485] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:14:59.043500 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:59.047910 30200 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:14:59.067711 29809 consensus_peers.cc:485] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:14:59.068969 30070 consensus_peers.cc:485] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:14:59.123196 371 consensus_peers.cc:485] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:14:59.207989 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:14:59.234815 16990 ts_tablet_manager.cc:902] T 2baf70e0e1894cea9fdeb92279bc0cff P f575ce92c0224eeb80b9fae5f6561fbe: Initiating tablet copy from peer fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:14:59.236208 16990 tablet_copy_client.cc:308] T 2baf70e0e1894cea9fdeb92279bc0cff P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.7:44241
I20240503 02:14:59.236850 457 tablet_copy_service.cc:140] P fbbc2be6382e47908ecb7c68c367bd26: Received BeginTabletCopySession request for tablet 2baf70e0e1894cea9fdeb92279bc0cff from peer f575ce92c0224eeb80b9fae5f6561fbe ({username='slave'} at 127.22.108.8:48093)
I20240503 02:14:59.236997 457 tablet_copy_service.cc:161] P fbbc2be6382e47908ecb7c68c367bd26: Beginning new tablet copy session on tablet 2baf70e0e1894cea9fdeb92279bc0cff from peer f575ce92c0224eeb80b9fae5f6561fbe at {username='slave'} at 127.22.108.8:48093: session id = f575ce92c0224eeb80b9fae5f6561fbe-2baf70e0e1894cea9fdeb92279bc0cff
I20240503 02:14:59.239261 457 tablet_copy_source_session.cc:205] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:59.240301 16990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2baf70e0e1894cea9fdeb92279bc0cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:59.245218 16990 tablet_copy_client.cc:783] T 2baf70e0e1894cea9fdeb92279bc0cff P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:59.245446 16990 tablet_copy_client.cc:647] T 2baf70e0e1894cea9fdeb92279bc0cff P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:59.247189 16990 tablet_copy_client.cc:523] T 2baf70e0e1894cea9fdeb92279bc0cff P f575ce92c0224eeb80b9fae5f6561fbe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:59.249889 16990 tablet_bootstrap.cc:492] T 2baf70e0e1894cea9fdeb92279bc0cff P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap starting.
I20240503 02:14:59.257687 16990 tablet_bootstrap.cc:492] T 2baf70e0e1894cea9fdeb92279bc0cff P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:59.258201 16990 tablet_bootstrap.cc:492] T 2baf70e0e1894cea9fdeb92279bc0cff P f575ce92c0224eeb80b9fae5f6561fbe: Bootstrap complete.
I20240503 02:14:59.258500 16990 ts_tablet_manager.cc:1367] T 2baf70e0e1894cea9fdeb92279bc0cff P f575ce92c0224eeb80b9fae5f6561fbe: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20240503 02:14:59.259095 16990 raft_consensus.cc:348] T 2baf70e0e1894cea9fdeb92279bc0cff P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:14:59.259259 16990 raft_consensus.cc:729] T 2baf70e0e1894cea9fdeb92279bc0cff P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f575ce92c0224eeb80b9fae5f6561fbe, State: Initialized, Role: FOLLOWER
I20240503 02:14:59.259447 16990 consensus_queue.cc:260] T 2baf70e0e1894cea9fdeb92279bc0cff P f575ce92c0224eeb80b9fae5f6561fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:14:59.259889 16990 ts_tablet_manager.cc:1398] T 2baf70e0e1894cea9fdeb92279bc0cff P f575ce92c0224eeb80b9fae5f6561fbe: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:14:59.260637 457 tablet_copy_service.cc:342] P fbbc2be6382e47908ecb7c68c367bd26: Request end of tablet copy session f575ce92c0224eeb80b9fae5f6561fbe-2baf70e0e1894cea9fdeb92279bc0cff received from {username='slave'} at 127.22.108.8:48093
I20240503 02:14:59.260762 457 tablet_copy_service.cc:433] P fbbc2be6382e47908ecb7c68c367bd26: ending tablet copy session f575ce92c0224eeb80b9fae5f6561fbe-2baf70e0e1894cea9fdeb92279bc0cff on tablet 2baf70e0e1894cea9fdeb92279bc0cff with peer f575ce92c0224eeb80b9fae5f6561fbe
W20240503 02:14:59.343261 30200 consensus_peers.cc:485] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:14:59.393692 502 consensus_peers.cc:485] T ba97049157a646c780fa203d34cd4c82 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:14:59.413328 16998 ts_tablet_manager.cc:902] T 9b3b9d83e4c742bfa62624676cbb6a11 P 1f10f80f13c84e80b03eda7085a3b012: Initiating tablet copy from peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:14:59.414770 16998 tablet_copy_client.cc:308] T 9b3b9d83e4c742bfa62624676cbb6a11 P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.3:42679
I20240503 02:14:59.415403 30156 tablet_copy_service.cc:140] P a9094240860e43bfb129193f3bfd5be6: Received BeginTabletCopySession request for tablet 9b3b9d83e4c742bfa62624676cbb6a11 from peer 1f10f80f13c84e80b03eda7085a3b012 ({username='slave'} at 127.22.108.4:33331)
I20240503 02:14:59.415555 30156 tablet_copy_service.cc:161] P a9094240860e43bfb129193f3bfd5be6: Beginning new tablet copy session on tablet 9b3b9d83e4c742bfa62624676cbb6a11 from peer 1f10f80f13c84e80b03eda7085a3b012 at {username='slave'} at 127.22.108.4:33331: session id = 1f10f80f13c84e80b03eda7085a3b012-9b3b9d83e4c742bfa62624676cbb6a11
I20240503 02:14:59.417831 30156 tablet_copy_source_session.cc:205] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:59.419270 16998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b3b9d83e4c742bfa62624676cbb6a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:59.425192 16998 tablet_copy_client.cc:783] T 9b3b9d83e4c742bfa62624676cbb6a11 P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:59.425408 16998 tablet_copy_client.cc:647] T 9b3b9d83e4c742bfa62624676cbb6a11 P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:59.427652 16998 tablet_copy_client.cc:523] T 9b3b9d83e4c742bfa62624676cbb6a11 P 1f10f80f13c84e80b03eda7085a3b012: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:59.430897 16998 tablet_bootstrap.cc:492] T 9b3b9d83e4c742bfa62624676cbb6a11 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap starting.
W20240503 02:14:59.435411 29809 consensus_peers.cc:485] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:14:59.436499 502 consensus_peers.cc:485] T 36d3f1edbab7477f9b7ac400160491a0 P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:14:59.438742 16998 tablet_bootstrap.cc:492] T 9b3b9d83e4c742bfa62624676cbb6a11 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 02:14:59.439090 16998 tablet_bootstrap.cc:492] T 9b3b9d83e4c742bfa62624676cbb6a11 P 1f10f80f13c84e80b03eda7085a3b012: Bootstrap complete.
I20240503 02:14:59.439265 16998 ts_tablet_manager.cc:1367] T 9b3b9d83e4c742bfa62624676cbb6a11 P 1f10f80f13c84e80b03eda7085a3b012: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20240503 02:14:59.439836 16998 raft_consensus.cc:348] T 9b3b9d83e4c742bfa62624676cbb6a11 P 1f10f80f13c84e80b03eda7085a3b012 [term 2 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:14:59.440301 16998 raft_consensus.cc:729] T 9b3b9d83e4c742bfa62624676cbb6a11 P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f10f80f13c84e80b03eda7085a3b012, State: Initialized, Role: FOLLOWER
I20240503 02:14:59.440472 16998 consensus_queue.cc:260] T 9b3b9d83e4c742bfa62624676cbb6a11 P 1f10f80f13c84e80b03eda7085a3b012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:14:59.440938 16998 ts_tablet_manager.cc:1398] T 9b3b9d83e4c742bfa62624676cbb6a11 P 1f10f80f13c84e80b03eda7085a3b012: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:14:59.442421 30156 tablet_copy_service.cc:342] P a9094240860e43bfb129193f3bfd5be6: Request end of tablet copy session 1f10f80f13c84e80b03eda7085a3b012-9b3b9d83e4c742bfa62624676cbb6a11 received from {username='slave'} at 127.22.108.4:33331
I20240503 02:14:59.442533 30156 tablet_copy_service.cc:433] P a9094240860e43bfb129193f3bfd5be6: ending tablet copy session 1f10f80f13c84e80b03eda7085a3b012-9b3b9d83e4c742bfa62624676cbb6a11 on tablet 9b3b9d83e4c742bfa62624676cbb6a11 with peer 1f10f80f13c84e80b03eda7085a3b012
W20240503 02:14:59.501199 371 consensus_peers.cc:485] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:14:59.584033 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:59.591284 371 consensus_peers.cc:485] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:14:59.621372 565 raft_consensus.cc:1203] T 2baf70e0e1894cea9fdeb92279bc0cff P f575ce92c0224eeb80b9fae5f6561fbe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240503 02:14:59.621848 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:14:59.624493 29940 consensus_peers.cc:485] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:14:59.633634 502 consensus_peers.cc:485] T 9be5395489a3488da62f4d5b66facf3d P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:14:59.710348 29940 consensus_peers.cc:485] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:14:59.747521 30070 consensus_peers.cc:485] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:14:59.768018 30200 consensus_peers.cc:485] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:14:59.793298 30265 raft_consensus.cc:1203] T 9b3b9d83e4c742bfa62624676cbb6a11 P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.4->[2.5-2.5] Dedup: 2.5->[]
I20240503 02:14:59.794142 16927 raft_consensus.cc:2932] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:14:59.794965 30265 raft_consensus.cc:2932] T 9b3b9d83e4c742bfa62624676cbb6a11 P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:14:59.806406 307 raft_consensus.cc:2932] T 9b3b9d83e4c742bfa62624676cbb6a11 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:14:59.809346 29726 catalog_manager.cc:5484] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 reported cstate change: config changed from index 3 to 5, VOTER 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4) added. New cstate: current_term: 2 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
W20240503 02:14:59.824167 30200 consensus_peers.cc:485] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:14:59.828545 371 consensus_peers.cc:485] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:14:59.872115 307 consensus_queue.cc:237] T ebcd2c6e1722442fbe603cfcccb56ab0 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:14:59.878942 435 raft_consensus.cc:1261] T ebcd2c6e1722442fbe603cfcccb56ab0 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:14:59.880069 32709 consensus_peers.cc:485] T ebcd2c6e1722442fbe603cfcccb56ab0 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Couldn't send request to peer 155735c7f8b14997ac4c770aee2a9401. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ebcd2c6e1722442fbe603cfcccb56ab0. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:59.880177 16997 consensus_queue.cc:1035] T ebcd2c6e1722442fbe603cfcccb56ab0 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:59.883397 30134 raft_consensus.cc:1261] T ebcd2c6e1722442fbe603cfcccb56ab0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:14:59.884433 16997 consensus_queue.cc:1035] T ebcd2c6e1722442fbe603cfcccb56ab0 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:14:59.889072 16571 raft_consensus.cc:2932] T ebcd2c6e1722442fbe603cfcccb56ab0 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:14:59.895095 30134 raft_consensus.cc:2932] T ebcd2c6e1722442fbe603cfcccb56ab0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:14:59.902273 29727 catalog_manager.cc:5484] T ebcd2c6e1722442fbe603cfcccb56ab0 P a9094240860e43bfb129193f3bfd5be6 reported cstate change: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New cstate: current_term: 1 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:14:59.904982 435 raft_consensus.cc:2932] T ebcd2c6e1722442fbe603cfcccb56ab0 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } }
I20240503 02:14:59.937275 30265 consensus_queue.cc:237] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
W20240503 02:14:59.944646 30200 consensus_peers.cc:485] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:14:59.944646 30199 consensus_peers.cc:485] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6bf84757d1fb44689da830a82e857787. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:59.945492 30134 raft_consensus.cc:1261] T 6bf84757d1fb44689da830a82e857787 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Refusing update from remote peer 1f10f80f13c84e80b03eda7085a3b012: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:14:59.946084 16941 consensus_queue.cc:1035] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 02:14:59.999087 502 consensus_peers.cc:485] T 6bd4fcc4faf44d148a03beaafc1c05ba P f575ce92c0224eeb80b9fae5f6561fbe -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:00.069494 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:00.087648 29809 consensus_peers.cc:485] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:00.088812 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:00.108479 30070 consensus_peers.cc:485] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:00.167770 30070 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:00.174268 30070 consensus_peers.cc:485] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:00.250109 30070 consensus_peers.cc:485] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:15:00.307066 483 maintenance_manager.cc:419] P fbbc2be6382e47908ecb7c68c367bd26: Scheduling FlushMRSOp(35a5a7f4fe184b9fadc94c98b61deaf0): perf score=0.033367
W20240503 02:15:00.311177 32709 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:00.319304 30200 consensus_peers.cc:485] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:00.321012 32709 consensus_peers.cc:485] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:15:00.371241 374 maintenance_manager.cc:643] P fbbc2be6382e47908ecb7c68c367bd26: FlushMRSOp(35a5a7f4fe184b9fadc94c98b61deaf0) complete. Timing: real 0.064s user 0.040s sys 0.019s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":486,"dirs.queue_time_us":6416,"dirs.run_cpu_time_us":230,"dirs.run_wall_time_us":916,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":656,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":7363,"threads_started":3,"wal-append.queue_time_us":485}
I20240503 02:15:00.374147 483 maintenance_manager.cc:419] P fbbc2be6382e47908ecb7c68c367bd26: Scheduling UndoDeltaBlockGCOp(35a5a7f4fe184b9fadc94c98b61deaf0): 4923 bytes on disk
I20240503 02:15:00.374775 374 maintenance_manager.cc:643] P fbbc2be6382e47908ecb7c68c367bd26: UndoDeltaBlockGCOp(35a5a7f4fe184b9fadc94c98b61deaf0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
I20240503 02:15:00.386498 17055 ts_tablet_manager.cc:902] T 6bf84757d1fb44689da830a82e857787 P f575ce92c0224eeb80b9fae5f6561fbe: Initiating tablet copy from peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
F20240503 02:15:00.387806 17055 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-1c9200fa969a46e6a48c64293801382b.queue_time_us, tablet-download-pool-1c9200fa969a46e6a48c64293801382b.queue_time_us) (tablet-download-pool-1c9200fa969a46e6a48c64293801382b.run_cpu_time_us, tablet-download-pool-1c9200fa969a46e6a48c64293801382b.run_cpu_time_us) (tablet-download-pool-1c9200fa969a46e6a48c64293801382b.run_wall_time_us, tablet-download-pool-1c9200fa969a46e6a48c64293801382b.run_wall_time_us) (tablet-download-pool-2baf70e0e1894cea9fdeb92279bc0cff.queue_time_us, tablet-download-pool-2baf70e0e1894cea9fdeb92279bc0cff.queue_time_us) (tablet-download-pool-2baf70e0e1894cea9fdeb92279bc0cff.run_cpu_time_us, tablet-download-pool-2baf70e0e1894cea9fdeb92279bc0cff.run_cpu_time_us) (tablet-download-pool-2baf70e0e1894cea9fdeb92279bc0cff.run_wall_time_us, tablet-download-pool-2baf70e0e1894cea9fdeb92279bc0cff.run_wall_time_us) (tablet-download-pool-2bf8c34ee16d4e0fbc6817d63b2ba52e.queue_time_us, tablet-download-pool-2bf8c34ee16d4e0fbc6817d63b2ba52e.queue_time_us) (tablet-download-pool-2bf8c34ee16d4e0fbc6817d63b2ba52e.run_cpu_time_us, tablet-download-pool-2bf8c34ee16d4e0fbc6817d63b2ba52e.run_cpu_time_us) (tablet-download-pool-2bf8c34ee16d4e0fbc6817d63b2ba52e.run_wall_time_us, tablet-download-pool-2bf8c34ee16d4e0fbc6817d63b2ba52e.run_wall_time_us) (tablet-download-pool-46d608d84863424e8c65feac7c2e835f.queue_time_us, tablet-download-pool-46d608d84863424e8c65feac7c2e835f.queue_time_us) (tablet-download-pool-46d608d84863424e8c65feac7c2e835f.run_cpu_time_us, tablet-download-pool-46d608d84863424e8c65feac7c2e835f.run_cpu_time_us) (tablet-download-pool-46d608d84863424e8c65feac7c2e835f.run_wall_time_us, tablet-download-pool-46d608d84863424e8c65feac7c2e835f.run_wall_time_us) (tablet-download-pool-5040203bd48549ff9c8f64c71d6bc598.queue_time_us, tablet-download-pool-5040203bd48549ff9c8f64c71d6bc598.queue_time_us) (tablet-download-pool-5040203bd48549ff9c8f64c71d6bc598.run_cpu_time_us, tablet-download-pool-5040203bd48549ff9c8f64c71d6bc598.run_cpu_time_us) (tablet-download-pool-5040203bd48549ff9c8f64c71d6bc598.run_wall_time_us, tablet-download-pool-5040203bd48549ff9c8f64c71d6bc598.run_wall_time_us) (tablet-download-pool-58e1aaecf47944569296fc1e0780efb5.queue_time_us, tablet-download-pool-58e1aaecf47944569296fc1e0780efb5.queue_time_us) (tablet-download-pool-58e1aaecf47944569296fc1e0780efb5.run_cpu_time_us, tablet-download-pool-58e1aaecf47944569296fc1e0780efb5.run_cpu_time_us) (tablet-download-pool-58e1aaecf47944569296fc1e0780efb5.run_wall_time_us, tablet-download-pool-58e1aaecf47944569296fc1e0780efb5.run_wall_time_us) (tablet-download-pool-5faccb32e55d4cb689ff491fd20884e6.queue_time_us, tablet-download-pool-5faccb32e55d4cb689ff491fd20884e6.queue_time_us) (tablet-download-pool-5faccb32e55d4cb689ff491fd20884e6.run_cpu_time_us, tablet-download-pool-5faccb32e55d4cb689ff491fd20884e6.run_cpu_time_us) (tablet-download-pool-5faccb32e55d4cb689ff491fd20884e6.run_wall_time_us, tablet-download-pool-5faccb32e55d4cb689ff491fd20884e6.run_wall_time_us) (tablet-download-pool-6bf84757d1fb44689da830a82e857787.queue_time_us, tablet-download-pool-6bf84757d1fb44689da830a82e857787.queue_time_us) (tablet-download-pool-7527cc5931e24fa180efedbf81438efd.queue_time_us, tablet-download-pool-7527cc5931e24fa180efedbf81438efd.queue_time_us) (tablet-download-pool-7527cc5931e24fa180efedbf81438efd.run_cpu_time_us, tablet-download-pool-7527cc5931e24fa180efedbf81438efd.run_cpu_time_us) (tablet-download-pool-7527cc5931e24fa180efedbf81438efd.run_wall_time_us, tablet-download-pool-7527cc5931e24fa180efedbf81438efd.run_wall_time_us) (tablet-download-pool-779a6685fc8c4670a71b51c4493bc2f7.queue_time_us, tablet-download-pool-779a6685fc8c4670a71b51c4493bc2f7.queue_time_us) (tablet-download-pool-779a6685fc8c4670a71b51c4493bc2f7.run_cpu_time_us, tablet-download-pool-779a6685fc8c4670a71b51c4493bc2f7.run_cpu_time_us) (tablet-download-pool-779a6685fc8c4670a71b51c4493bc2f7.run_wall_time_us, tablet-download-pool-779a6685fc8c4670a71b51c4493bc2f7.run_wall_time_us) (tablet-download-pool-88b57085de3144cea3f64922ae5ccf55.queue_time_us, tablet-download-pool-88b57085de3144cea3f64922ae5ccf55.queue_time_us) (tablet-download-pool-88b57085de3144cea3f64922ae5ccf55.run_cpu_time_us, tablet-download-pool-88b57085de3144cea3f64922ae5ccf55.run_cpu_time_us) (tablet-download-pool-88b57085de3144cea3f64922ae5ccf55.run_wall_time_us, tablet-download-pool-88b57085de3144cea3f64922ae5ccf55.run_wall_time_us) (tablet-download-pool-c19017f829a74cc2b6dc41a2a25a3831.queue_time_us, tablet-download-pool-c19017f829a74cc2b6dc41a2a25a3831.queue_time_us) (tablet-download-pool-c19017f829a74cc2b6dc41a2a25a3831.run_cpu_time_us, tablet-download-pool-c19017f829a74cc2b6dc41a2a25a3831.run_cpu_time_us) (tablet-download-pool-c19017f829a74cc2b6dc41a2a25a3831.run_wall_time_us, tablet-download-pool-c19017f829a74cc2b6dc41a2a25a3831.run_wall_time_us) (tablet-download-pool-f694fefb42d84318a6d91532105b3c90.queue_time_us, tablet-download-pool-f694fefb42d84318a6d91532105b3c90.queue_time_us) (tablet-download-pool-f694fefb42d84318a6d91532105b3c90.run_cpu_time_us, tablet-download-pool-f694fefb42d84318a6d91532105b3c90.run_cpu_time_us) (tablet-download-pool-f694fefb42d84318a6d91532105b3c90.run_wall_time_us, tablet-download-pool-f694fefb42d84318a6d91532105b3c90.run_wall_time_us) (tablet-download-pool-f82e505ac13a41019e422b9dc1e4fef9.queue_time_us, tablet-download-pool-f82e505ac13a41019e422b9dc1e4fef9.queue_time_us) (tablet-download-pool-f82e505ac13a41019e422b9dc1e4fef9.run_cpu_time_us, tablet-download-pool-f82e505ac13a41019e422b9dc1e4fef9.run_cpu_time_us) (tablet-download-pool-f82e505ac13a41019e422b9dc1e4fef9.run_wall_time_us, tablet-download-pool-f82e505ac13a41019e422b9dc1e4fef9.run_wall_time_us) (tablet-download-pool-fa288463ae154e28afd9c71b4b55ef3a.queue_time_us, tablet-download-pool-fa288463ae154e28afd9c71b4b55ef3a.queue_time_us) (tablet-download-pool-fa288463ae154e28afd9c71b4b55ef3a.run_cpu_time_us, tablet-download-pool-fa288463ae154e28afd9c71b4b55ef3a.run_cpu_time_us) (tablet-download-pool-fa288463ae154e28afd9c71b4b55ef3a.run_wall_time_us, tablet-download-pool-fa288463ae154e28afd9c71b4b55ef3a.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7fa9c6ea8dcd google::LogMessage::Fail() at ??:0
W20240503 02:15:00.394165 30070 consensus_peers.cc:485] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
@ 0x7fa9c6eacb93 google::LogMessage::SendToLog() at ??:0
@ 0x7fa9c6ea87cc google::LogMessage::Flush() at ??:0
@ 0x7fa9c6ea9f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa9c9be9742 kudu::TraceMetrics::InternName() at ??:0
I20240503 02:15:00.417420 17057 ts_tablet_manager.cc:902] T ebcd2c6e1722442fbe603cfcccb56ab0 P 155735c7f8b14997ac4c770aee2a9401: Initiating tablet copy from peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
@ 0x7fa9c9bb5f8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fa9c9bafcd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fa9d90038de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240503 02:15:00.431190 17057 tablet_copy_client.cc:308] T ebcd2c6e1722442fbe603cfcccb56ab0 P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Beginning tablet copy session from remote peer at address 127.22.108.6:39975
I20240503 02:15:00.431866 327 tablet_copy_service.cc:140] P 8a666e798fe8477d8dd1e51b385d5ead: Received BeginTabletCopySession request for tablet ebcd2c6e1722442fbe603cfcccb56ab0 from peer 155735c7f8b14997ac4c770aee2a9401 ({username='slave'} at 127.22.108.2:53863)
I20240503 02:15:00.432062 327 tablet_copy_service.cc:161] P 8a666e798fe8477d8dd1e51b385d5ead: Beginning new tablet copy session on tablet ebcd2c6e1722442fbe603cfcccb56ab0 from peer 155735c7f8b14997ac4c770aee2a9401 at {username='slave'} at 127.22.108.2:53863: session id = 155735c7f8b14997ac4c770aee2a9401-ebcd2c6e1722442fbe603cfcccb56ab0
I20240503 02:15:00.435652 327 tablet_copy_source_session.cc:205] T ebcd2c6e1722442fbe603cfcccb56ab0 P 8a666e798fe8477d8dd1e51b385d5ead: Tablet Copy: opened 0 blocks and 1 log segments
@ 0x7fa9d901bdaf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240503 02:15:00.437108 17057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebcd2c6e1722442fbe603cfcccb56ab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:00.443598 17057 tablet_copy_client.cc:783] T ebcd2c6e1722442fbe603cfcccb56ab0 P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:00.443866 17057 tablet_copy_client.cc:647] T ebcd2c6e1722442fbe603cfcccb56ab0 P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:00.448031 17057 tablet_copy_client.cc:523] T ebcd2c6e1722442fbe603cfcccb56ab0 P 155735c7f8b14997ac4c770aee2a9401: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7fa9d9146480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240503 02:15:00.451514 17057 tablet_bootstrap.cc:492] T ebcd2c6e1722442fbe603cfcccb56ab0 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap starting.
@ 0x7fa9d917ea8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240503 02:15:00.470826 17057 tablet_bootstrap.cc:492] T ebcd2c6e1722442fbe603cfcccb56ab0 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:00.471112 17057 tablet_bootstrap.cc:492] T ebcd2c6e1722442fbe603cfcccb56ab0 P 155735c7f8b14997ac4c770aee2a9401: Bootstrap complete.
I20240503 02:15:00.471311 17057 ts_tablet_manager.cc:1367] T ebcd2c6e1722442fbe603cfcccb56ab0 P 155735c7f8b14997ac4c770aee2a9401: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.004s
I20240503 02:15:00.471871 17057 raft_consensus.cc:348] T ebcd2c6e1722442fbe603cfcccb56ab0 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:15:00.472067 17057 raft_consensus.cc:729] T ebcd2c6e1722442fbe603cfcccb56ab0 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Initialized, Role: FOLLOWER
I20240503 02:15:00.472281 17057 consensus_queue.cc:260] T ebcd2c6e1722442fbe603cfcccb56ab0 P 155735c7f8b14997ac4c770aee2a9401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } }
I20240503 02:15:00.472800 17057 ts_tablet_manager.cc:1398] T ebcd2c6e1722442fbe603cfcccb56ab0 P 155735c7f8b14997ac4c770aee2a9401: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
@ 0x7fa9d9161981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7fa9d916170e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
W20240503 02:15:00.490998 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:00.491271 327 tablet_copy_service.cc:342] P 8a666e798fe8477d8dd1e51b385d5ead: Request end of tablet copy session 155735c7f8b14997ac4c770aee2a9401-ebcd2c6e1722442fbe603cfcccb56ab0 received from {username='slave'} at 127.22.108.2:53863
I20240503 02:15:00.491415 327 tablet_copy_service.cc:433] P 8a666e798fe8477d8dd1e51b385d5ead: ending tablet copy session 155735c7f8b14997ac4c770aee2a9401-ebcd2c6e1722442fbe603cfcccb56ab0 on tablet ebcd2c6e1722442fbe603cfcccb56ab0 with peer 155735c7f8b14997ac4c770aee2a9401
@ 0x7fa9c9bb9b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa9c9bbd19c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fa9c9bbd019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fa9c9b9801e kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa9ccf1d6db start_thread at ??:0
@ 0x7fa9c5d5b71f clone at ??:0
W20240503 02:15:00.563544 29940 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:00.630683 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:00.746393 29809 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:15:00.784442 30004 raft_consensus.cc:1203] T ebcd2c6e1722442fbe603cfcccb56ab0 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240503 02:15:00.801355 29809 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:00.819983 30069 connection.cc:535] client connection to 127.22.108.8:42233 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:15:00.820243 30199 connection.cc:535] client connection to 127.22.108.8:42233 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:15:00.820637 29808 connection.cc:535] client connection to 127.22.108.8:42233 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:15:00.821000 32708 connection.cc:535] client connection to 127.22.108.8:42233 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:15:00.822583 370 connection.cc:535] client connection to 127.22.108.8:42233 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:15:00.822711 32708 consensus_peers.cc:485] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:15:00.822849 32708 consensus_peers.cc:485] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:15:00.822948 32708 consensus_peers.cc:485] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:15:00.823062 32708 consensus_peers.cc:485] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:15:00.823405 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:00.823532 30199 consensus_peers.cc:485] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:15:00.823627 30199 consensus_peers.cc:485] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:15:00.823710 30199 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:15:00.823782 30199 consensus_peers.cc:485] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:15:00.823849 30199 consensus_peers.cc:485] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:15:00.823861 29808 consensus_peers.cc:485] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:15:00.824081 29808 consensus_peers.cc:485] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:15:00.825448 370 consensus_peers.cc:485] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:15:00.825552 370 consensus_peers.cc:485] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:15:00.826670 30069 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:15:00.826813 30069 consensus_peers.cc:485] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:15:00.826908 30069 consensus_peers.cc:485] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:15:00.827054 30069 consensus_peers.cc:485] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:15:00.829083 370 consensus_peers.cc:485] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:15:00.834252 29808 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:15:00.835525 29939 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:15:00.837405 29808 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:15:00.840355 30069 consensus_peers.cc:485] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:15:00.842377 370 consensus_peers.cc:485] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:15:00.851735 30199 consensus_peers.cc:485] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:15:00.852324 32708 consensus_peers.cc:485] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:15:00.865020 370 consensus_peers.cc:485] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:15:00.887691 370 consensus_peers.cc:485] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:15:00.888563 29808 consensus_peers.cc:485] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:15:00.895750 30199 consensus_peers.cc:485] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:15:00.943639 32708 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:15:00.963294 29940 consensus_peers.cc:485] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:00.966459 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:00.986382 30199 consensus_peers.cc:485] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:15:00.990042 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:01.002517 29808 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:15:01.006994 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:01.020275 30199 consensus_peers.cc:485] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:15:01.055509 29809 consensus_peers.cc:485] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:01.058485 29809 consensus_peers.cc:485] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:01.172896 371 consensus_peers.cc:485] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:01.185947 30070 consensus_peers.cc:485] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:01.237514 32709 consensus_peers.cc:485] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:01.273170 30070 consensus_peers.cc:485] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:15:01.332031 29809 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:01.411414 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:01.437105 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:01.447994 29809 consensus_peers.cc:485] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:01.510776 30200 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:15:01.517097 17006 raft_consensus.cc:482] T 4563386631774e74a4b12fc71714c1a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:01.517302 17006 raft_consensus.cc:504] T 4563386631774e74a4b12fc71714c1a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:15:01.518066 17006 leader_election.cc:288] T 4563386631774e74a4b12fc71714c1a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:15:01.519162 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4563386631774e74a4b12fc71714c1a0" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "155735c7f8b14997ac4c770aee2a9401" is_pre_election: true
W20240503 02:15:01.519240 30069 leader_election.cc:334] T 4563386631774e74a4b12fc71714c1a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:01.519721 30070 leader_election.cc:302] T 4563386631774e74a4b12fc71714c1a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6; no voters: 155735c7f8b14997ac4c770aee2a9401, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:01.521137 17006 raft_consensus.cc:2726] T 4563386631774e74a4b12fc71714c1a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:15:01.537480 17029 raft_consensus.cc:482] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:01.537635 17029 raft_consensus.cc:504] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:15:01.538249 17029 leader_election.cc:288] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
W20240503 02:15:01.539742 32709 leader_election.cc:334] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:01.539906 32708 leader_election.cc:334] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:01.540051 32708 leader_election.cc:302] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:01.540306 17029 raft_consensus.cc:2726] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:15:01.552497 30070 consensus_peers.cc:485] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:15:01.567432 17006 raft_consensus.cc:482] T 7527cc5931e24fa180efedbf81438efd P a9094240860e43bfb129193f3bfd5be6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:01.567626 17006 raft_consensus.cc:504] T 7527cc5931e24fa180efedbf81438efd P a9094240860e43bfb129193f3bfd5be6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:15:01.568298 17006 leader_election.cc:288] T 7527cc5931e24fa180efedbf81438efd P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:15:01.568681 29874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7527cc5931e24fa180efedbf81438efd" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
W20240503 02:15:01.569733 30069 leader_election.cc:334] T 7527cc5931e24fa180efedbf81438efd P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:01.569844 30069 leader_election.cc:302] T 7527cc5931e24fa180efedbf81438efd P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6; no voters: 16440e365430405fb10366ab70e2b429, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:01.570132 17006 raft_consensus.cc:2726] T 7527cc5931e24fa180efedbf81438efd P a9094240860e43bfb129193f3bfd5be6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:15:01.586606 353 maintenance_manager.cc:419] P 8a666e798fe8477d8dd1e51b385d5ead: Scheduling FlushMRSOp(5f4b72b138c64ac58e6477cd2b1fb215): perf score=0.033372
I20240503 02:15:01.590978 16399 raft_consensus.cc:482] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:01.591145 16399 raft_consensus.cc:504] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:15:01.592372 16399 leader_election.cc:288] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:15:01.593382 370 leader_election.cc:334] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:01.594380 371 leader_election.cc:334] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:01.594507 371 leader_election.cc:302] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:01.595209 16399 raft_consensus.cc:2726] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:01.606962 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:01.638702 483 maintenance_manager.cc:419] P fbbc2be6382e47908ecb7c68c367bd26: Scheduling FlushMRSOp(96de1c26ae884ea18b63b328db040ba1): perf score=0.033391
I20240503 02:15:01.648669 17087 raft_consensus.cc:482] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:01.649096 17087 raft_consensus.cc:504] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:15:01.649943 17087 leader_election.cc:288] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:15:01.656200 370 leader_election.cc:334] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:01.659309 371 leader_election.cc:334] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:01.659478 371 leader_election.cc:302] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:01.660733 17087 raft_consensus.cc:2726] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:15:01.685804 32712 maintenance_manager.cc:643] P 8a666e798fe8477d8dd1e51b385d5ead: FlushMRSOp(5f4b72b138c64ac58e6477cd2b1fb215) complete. Timing: real 0.099s user 0.075s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":325,"dirs.queue_time_us":6731,"dirs.run_cpu_time_us":227,"dirs.run_wall_time_us":8829,"drs_written":1,"lbm_read_time_us":122,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1614,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":7324,"threads_started":3,"wal-append.queue_time_us":338}
I20240503 02:15:01.687780 353 maintenance_manager.cc:419] P 8a666e798fe8477d8dd1e51b385d5ead: Scheduling UndoDeltaBlockGCOp(5f4b72b138c64ac58e6477cd2b1fb215): 4923 bytes on disk
I20240503 02:15:01.688421 32712 maintenance_manager.cc:643] P 8a666e798fe8477d8dd1e51b385d5ead: UndoDeltaBlockGCOp(5f4b72b138c64ac58e6477cd2b1fb215) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240503 02:15:01.690529 353 maintenance_manager.cc:419] P 8a666e798fe8477d8dd1e51b385d5ead: Scheduling FlushMRSOp(08ee37b160d441ddabb640be907d6771): perf score=0.033334
I20240503 02:15:01.725741 374 maintenance_manager.cc:643] P fbbc2be6382e47908ecb7c68c367bd26: FlushMRSOp(96de1c26ae884ea18b63b328db040ba1) complete. Timing: real 0.087s user 0.064s sys 0.006s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":3506,"dirs.run_cpu_time_us":224,"dirs.run_wall_time_us":907,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":928,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":5144,"threads_started":2,"wal-append.queue_time_us":1874}
I20240503 02:15:01.727871 483 maintenance_manager.cc:419] P fbbc2be6382e47908ecb7c68c367bd26: Scheduling UndoDeltaBlockGCOp(96de1c26ae884ea18b63b328db040ba1): 4581 bytes on disk
I20240503 02:15:01.727981 17006 raft_consensus.cc:482] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:01.728171 17006 raft_consensus.cc:504] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:15:01.728823 17006 leader_election.cc:288] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:15:01.728976 374 maintenance_manager.cc:643] P fbbc2be6382e47908ecb7c68c367bd26: UndoDeltaBlockGCOp(96de1c26ae884ea18b63b328db040ba1) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
I20240503 02:15:01.730710 32712 maintenance_manager.cc:643] P 8a666e798fe8477d8dd1e51b385d5ead: FlushMRSOp(08ee37b160d441ddabb640be907d6771) complete. Timing: real 0.040s user 0.031s sys 0.007s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":37,"dirs.run_cpu_time_us":158,"dirs.run_wall_time_us":817,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":712,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":260,"threads_started":1,"wal-append.queue_time_us":346}
W20240503 02:15:01.732183 30070 leader_election.cc:334] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:01.732422 30069 leader_election.cc:334] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:01.732524 30069 leader_election.cc:302] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:01.732957 17006 raft_consensus.cc:2726] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:15:01.733417 353 maintenance_manager.cc:419] P 8a666e798fe8477d8dd1e51b385d5ead: Scheduling UndoDeltaBlockGCOp(08ee37b160d441ddabb640be907d6771): 4583 bytes on disk
I20240503 02:15:01.733922 32712 maintenance_manager.cc:643] P 8a666e798fe8477d8dd1e51b385d5ead: UndoDeltaBlockGCOp(08ee37b160d441ddabb640be907d6771) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20240503 02:15:01.739054 16834 raft_consensus.cc:482] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:01.739221 16834 raft_consensus.cc:504] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:15:01.739884 16834 leader_election.cc:288] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:15:01.742091 29808 leader_election.cc:334] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:01.742319 29809 leader_election.cc:334] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:01.742406 29809 leader_election.cc:302] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 16440e365430405fb10366ab70e2b429; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:01.742759 16834 raft_consensus.cc:2726] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:15:01.751382 16834 raft_consensus.cc:482] T eb0f4fe6b8684f329725ac2f7d5428f2 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:01.751529 16834 raft_consensus.cc:504] T eb0f4fe6b8684f329725ac2f7d5428f2 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:15:01.752323 16834 leader_election.cc:288] T eb0f4fe6b8684f329725ac2f7d5428f2 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
W20240503 02:15:01.753695 371 consensus_peers.cc:485] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:15:01.753755 435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb0f4fe6b8684f329725ac2f7d5428f2" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
W20240503 02:15:01.755347 29808 leader_election.cc:334] T eb0f4fe6b8684f329725ac2f7d5428f2 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:01.755477 29808 leader_election.cc:302] T eb0f4fe6b8684f329725ac2f7d5428f2 P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 16440e365430405fb10366ab70e2b429; no voters: f575ce92c0224eeb80b9fae5f6561fbe, fbbc2be6382e47908ecb7c68c367bd26
I20240503 02:15:01.756304 16834 raft_consensus.cc:2726] T eb0f4fe6b8684f329725ac2f7d5428f2 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:15:01.778198 16834 raft_consensus.cc:482] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:01.778379 16834 raft_consensus.cc:504] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:15:01.779027 16834 leader_election.cc:288] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:15:01.779695 30135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7527cc5931e24fa180efedbf81438efd" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:15:01.779898 30135 raft_consensus.cc:2454] T 7527cc5931e24fa180efedbf81438efd P a9094240860e43bfb129193f3bfd5be6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 2.
I20240503 02:15:01.780272 29808 leader_election.cc:302] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, a9094240860e43bfb129193f3bfd5be6; no voters:
I20240503 02:15:01.780553 16834 raft_consensus.cc:2781] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 02:15:01.780663 16834 raft_consensus.cc:482] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:01.780794 16834 raft_consensus.cc:3037] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 [term 2 FOLLOWER]: Advancing to term 3
W20240503 02:15:01.781239 29808 leader_election.cc:334] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:01.783778 30200 consensus_peers.cc:485] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:15:01.784029 16834 raft_consensus.cc:504] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:15:01.785604 16834 leader_election.cc:288] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 3 election: Requested vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:15:01.785905 30135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7527cc5931e24fa180efedbf81438efd" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
I20240503 02:15:01.786121 30135 raft_consensus.cc:3037] T 7527cc5931e24fa180efedbf81438efd P a9094240860e43bfb129193f3bfd5be6 [term 2 FOLLOWER]: Advancing to term 3
W20240503 02:15:01.787251 29808 leader_election.cc:334] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:01.788722 30135 raft_consensus.cc:2454] T 7527cc5931e24fa180efedbf81438efd P a9094240860e43bfb129193f3bfd5be6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16440e365430405fb10366ab70e2b429 in term 3.
I20240503 02:15:01.789130 29808 leader_election.cc:302] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16440e365430405fb10366ab70e2b429, a9094240860e43bfb129193f3bfd5be6; no voters: f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:01.789489 16834 raft_consensus.cc:2781] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:15:01.789645 16834 raft_consensus.cc:686] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 [term 3 LEADER]: Becoming Leader. State: Replica: 16440e365430405fb10366ab70e2b429, State: Running, Role: LEADER
I20240503 02:15:01.789916 16834 consensus_queue.cc:237] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:15:01.795365 29725 catalog_manager.cc:5484] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 reported cstate change: term changed from 2 to 3, leader changed from f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) to 16440e365430405fb10366ab70e2b429 (127.22.108.1). New cstate: current_term: 3 leader_uuid: "16440e365430405fb10366ab70e2b429" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } }
I20240503 02:15:01.834225 16834 raft_consensus.cc:482] T c2d33db64ee8482aad840af219d60b8b P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:01.834451 16834 raft_consensus.cc:504] T c2d33db64ee8482aad840af219d60b8b P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:15:01.835142 16834 leader_election.cc:288] T c2d33db64ee8482aad840af219d60b8b P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:15:01.835830 30135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d33db64ee8482aad840af219d60b8b" candidate_uuid: "16440e365430405fb10366ab70e2b429" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:15:01.836395 17081 raft_consensus.cc:482] T 5800022ae50043ee8b1af1e134e86e6a P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:01.836551 17081 raft_consensus.cc:504] T 5800022ae50043ee8b1af1e134e86e6a P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:15:01.837360 17081 leader_election.cc:288] T 5800022ae50043ee8b1af1e134e86e6a P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:15:01.837539 16927 raft_consensus.cc:482] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:01.837697 16927 raft_consensus.cc:504] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:15:01.838253 16927 leader_election.cc:288] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:15:01.838649 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5800022ae50043ee8b1af1e134e86e6a" candidate_uuid: "1f10f80f13c84e80b03eda7085a3b012" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
W20240503 02:15:01.840001 29808 leader_election.cc:334] T c2d33db64ee8482aad840af219d60b8b P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:01.840131 29808 leader_election.cc:302] T c2d33db64ee8482aad840af219d60b8b P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 16440e365430405fb10366ab70e2b429; no voters: a9094240860e43bfb129193f3bfd5be6, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:01.841728 16834 raft_consensus.cc:2726] T c2d33db64ee8482aad840af219d60b8b P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:01.842859 30199 leader_election.cc:334] T 5800022ae50043ee8b1af1e134e86e6a P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:01.842962 30199 leader_election.cc:302] T 5800022ae50043ee8b1af1e134e86e6a P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012; no voters: 8a666e798fe8477d8dd1e51b385d5ead, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:01.843279 17081 raft_consensus.cc:2726] T 5800022ae50043ee8b1af1e134e86e6a P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:01.843956 30069 leader_election.cc:334] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:01.844194 30070 leader_election.cc:334] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:01.844306 30070 leader_election.cc:302] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:01.844909 17006 raft_consensus.cc:2726] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:15:01.857379 17088 raft_consensus.cc:482] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:01.857550 17088 raft_consensus.cc:504] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:15:01.858340 17088 leader_election.cc:288] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:15:01.859609 30135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4563386631774e74a4b12fc71714c1a0" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6" is_pre_election: true
I20240503 02:15:01.859856 30135 raft_consensus.cc:2454] T 4563386631774e74a4b12fc71714c1a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 1.
W20240503 02:15:01.859982 29939 leader_election.cc:334] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:01.862046 29939 leader_election.cc:302] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, a9094240860e43bfb129193f3bfd5be6; no voters: f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:01.862506 17088 raft_consensus.cc:2781] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:15:01.862576 17088 raft_consensus.cc:482] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:01.862634 17088 raft_consensus.cc:3037] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:15:01.865461 17088 raft_consensus.cc:504] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:15:01.865979 17088 leader_election.cc:288] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 2 election: Requested vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:15:01.867347 30134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4563386631774e74a4b12fc71714c1a0" candidate_uuid: "155735c7f8b14997ac4c770aee2a9401" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a9094240860e43bfb129193f3bfd5be6"
W20240503 02:15:01.867467 29939 leader_election.cc:334] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:01.867532 30134 raft_consensus.cc:3037] T 4563386631774e74a4b12fc71714c1a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:15:01.870266 30134 raft_consensus.cc:2454] T 4563386631774e74a4b12fc71714c1a0 P a9094240860e43bfb129193f3bfd5be6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 155735c7f8b14997ac4c770aee2a9401 in term 2.
I20240503 02:15:01.871065 29939 leader_election.cc:302] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 155735c7f8b14997ac4c770aee2a9401, a9094240860e43bfb129193f3bfd5be6; no voters: f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:01.871659 17082 raft_consensus.cc:2781] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:15:01.871820 17082 raft_consensus.cc:686] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 [term 2 LEADER]: Becoming Leader. State: Replica: 155735c7f8b14997ac4c770aee2a9401, State: Running, Role: LEADER
I20240503 02:15:01.872115 17082 consensus_queue.cc:237] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
W20240503 02:15:01.877130 371 consensus_peers.cc:485] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:01.877363 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:01.878104 29725 catalog_manager.cc:5484] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 reported cstate change: term changed from 1 to 2, leader changed from f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) to 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2). New cstate: current_term: 2 leader_uuid: "155735c7f8b14997ac4c770aee2a9401" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
I20240503 02:15:01.896629 17029 raft_consensus.cc:482] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:01.896837 17029 raft_consensus.cc:504] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:15:01.897626 17029 leader_election.cc:288] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:15:01.898926 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5800022ae50043ee8b1af1e134e86e6a" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:15:01.899142 30266 raft_consensus.cc:2454] T 5800022ae50043ee8b1af1e134e86e6a P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
W20240503 02:15:01.899649 32708 leader_election.cc:334] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:01.900772 32709 leader_election.cc:302] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, 8a666e798fe8477d8dd1e51b385d5ead; no voters: f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:01.901192 17030 raft_consensus.cc:2781] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:15:01.901304 17030 raft_consensus.cc:482] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting leader election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:01.901410 17030 raft_consensus.cc:3037] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:15:01.903630 17030 raft_consensus.cc:504] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:15:01.904152 17030 leader_election.cc:288] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 election: Requested vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
W20240503 02:15:01.905519 32708 leader_election.cc:334] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:01.906663 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5800022ae50043ee8b1af1e134e86e6a" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012"
I20240503 02:15:01.906847 30266 raft_consensus.cc:3037] T 5800022ae50043ee8b1af1e134e86e6a P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:15:01.909669 30266 raft_consensus.cc:2454] T 5800022ae50043ee8b1af1e134e86e6a P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 2.
I20240503 02:15:01.910434 32709 leader_election.cc:302] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f10f80f13c84e80b03eda7085a3b012, 8a666e798fe8477d8dd1e51b385d5ead; no voters: f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:01.910688 17030 raft_consensus.cc:2781] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:15:01.910856 17030 raft_consensus.cc:686] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead [term 2 LEADER]: Becoming Leader. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Running, Role: LEADER
I20240503 02:15:01.911316 17030 consensus_queue.cc:237] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:15:01.917425 29725 catalog_manager.cc:5484] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: term changed from 1 to 2, leader changed from f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) to 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6). New cstate: current_term: 2 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } }
I20240503 02:15:01.945292 17087 raft_consensus.cc:482] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:01.945474 17087 raft_consensus.cc:504] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:15:01.946345 17087 leader_election.cc:288] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:15:01.946779 29872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb0f4fe6b8684f329725ac2f7d5428f2" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:15:01.947021 29872 raft_consensus.cc:2454] T eb0f4fe6b8684f329725ac2f7d5428f2 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 1.
I20240503 02:15:01.947705 373 leader_election.cc:302] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16440e365430405fb10366ab70e2b429, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:15:01.948594 16399 raft_consensus.cc:2781] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:15:01.948697 16399 raft_consensus.cc:482] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:01.948794 16399 raft_consensus.cc:3037] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:15:01.949326 370 leader_election.cc:334] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:01.951531 16399 raft_consensus.cc:504] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:15:01.952098 16399 leader_election.cc:288] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 election: Requested vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:15:01.952368 29872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb0f4fe6b8684f329725ac2f7d5428f2" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:15:01.952558 29872 raft_consensus.cc:3037] T eb0f4fe6b8684f329725ac2f7d5428f2 P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:15:01.953892 370 leader_election.cc:334] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:01.955410 16927 raft_consensus.cc:482] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:01.955617 16927 raft_consensus.cc:504] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:15:01.956452 16927 leader_election.cc:288] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:15:01.957526 29872 raft_consensus.cc:2454] T eb0f4fe6b8684f329725ac2f7d5428f2 P 16440e365430405fb10366ab70e2b429 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 2.
W20240503 02:15:01.957597 30069 leader_election.cc:334] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:01.958341 29872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d33db64ee8482aad840af219d60b8b" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429" is_pre_election: true
I20240503 02:15:01.958922 29872 raft_consensus.cc:2454] T c2d33db64ee8482aad840af219d60b8b P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 1.
I20240503 02:15:01.959282 373 leader_election.cc:302] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16440e365430405fb10366ab70e2b429, fbbc2be6382e47908ecb7c68c367bd26; no voters: f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:01.959271 30072 leader_election.cc:302] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16440e365430405fb10366ab70e2b429, a9094240860e43bfb129193f3bfd5be6; no voters: f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:01.959694 16399 raft_consensus.cc:2781] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:15:01.959846 16399 raft_consensus.cc:686] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 [term 2 LEADER]: Becoming Leader. State: Replica: fbbc2be6382e47908ecb7c68c367bd26, State: Running, Role: LEADER
I20240503 02:15:01.960263 16927 raft_consensus.cc:2781] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:15:01.960141 16399 consensus_queue.cc:237] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:15:01.960435 16927 raft_consensus.cc:482] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:01.960572 16927 raft_consensus.cc:3037] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:15:01.965214 16927 raft_consensus.cc:504] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
W20240503 02:15:01.968901 30069 leader_election.cc:334] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:01.969980 16927 leader_election.cc:288] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 election: Requested vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:15:01.971398 29872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d33db64ee8482aad840af219d60b8b" candidate_uuid: "a9094240860e43bfb129193f3bfd5be6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "16440e365430405fb10366ab70e2b429"
I20240503 02:15:01.971657 29872 raft_consensus.cc:3037] T c2d33db64ee8482aad840af219d60b8b P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:15:01.973018 29725 catalog_manager.cc:5484] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 reported cstate change: term changed from 1 to 2, leader changed from f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) to fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7). New cstate: current_term: 2 leader_uuid: "fbbc2be6382e47908ecb7c68c367bd26" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
I20240503 02:15:01.975220 29872 raft_consensus.cc:2454] T c2d33db64ee8482aad840af219d60b8b P 16440e365430405fb10366ab70e2b429 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9094240860e43bfb129193f3bfd5be6 in term 2.
I20240503 02:15:01.975762 30072 leader_election.cc:302] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16440e365430405fb10366ab70e2b429, a9094240860e43bfb129193f3bfd5be6; no voters: f575ce92c0224eeb80b9fae5f6561fbe
W20240503 02:15:01.977279 371 consensus_peers.cc:485] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 02:15:01.978078 16927 raft_consensus.cc:2781] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:15:01.978487 16927 raft_consensus.cc:686] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 [term 2 LEADER]: Becoming Leader. State: Replica: a9094240860e43bfb129193f3bfd5be6, State: Running, Role: LEADER
I20240503 02:15:01.978715 16927 consensus_queue.cc:237] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:15:01.983026 29725 catalog_manager.cc:5484] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 reported cstate change: term changed from 1 to 2, leader changed from f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8) to a9094240860e43bfb129193f3bfd5be6 (127.22.108.3). New cstate: current_term: 2 leader_uuid: "a9094240860e43bfb129193f3bfd5be6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } }
W20240503 02:15:02.046267 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:02.061405 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:02.146368 29809 consensus_peers.cc:485] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:02.159682 29940 consensus_peers.cc:485] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:15:02.190024 30134 raft_consensus.cc:1261] T 7527cc5931e24fa180efedbf81438efd P a9094240860e43bfb129193f3bfd5be6 [term 3 FOLLOWER]: Refusing update from remote peer 16440e365430405fb10366ab70e2b429: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240503 02:15:02.190584 16938 consensus_queue.cc:1035] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 02:15:02.201359 29808 consensus_peers.cc:485] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:15:02.229662 29940 consensus_peers.cc:485] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:02.249711 30200 consensus_peers.cc:485] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:02.251574 29939 consensus_peers.cc:485] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:15:02.254330 30070 consensus_peers.cc:485] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:02.280454 30070 consensus_peers.cc:485] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 02:15:02.353989 30266 raft_consensus.cc:1261] T 5800022ae50043ee8b1af1e134e86e6a P 1f10f80f13c84e80b03eda7085a3b012 [term 2 FOLLOWER]: Refusing update from remote peer 8a666e798fe8477d8dd1e51b385d5ead: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:15:02.354580 17030 consensus_queue.cc:1035] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:15:02.359129 32708 consensus_peers.cc:485] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:15:02.362249 30200 consensus_peers.cc:485] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:15:02.384965 29872 raft_consensus.cc:1261] T eb0f4fe6b8684f329725ac2f7d5428f2 P 16440e365430405fb10366ab70e2b429 [term 2 FOLLOWER]: Refusing update from remote peer fbbc2be6382e47908ecb7c68c367bd26: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:15:02.385577 16399 consensus_queue.cc:1035] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:15:02.396673 371 consensus_peers.cc:485] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:02.397066 370 consensus_peers.cc:485] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:02.400815 30134 raft_consensus.cc:1261] T 4563386631774e74a4b12fc71714c1a0 P a9094240860e43bfb129193f3bfd5be6 [term 2 FOLLOWER]: Refusing update from remote peer 155735c7f8b14997ac4c770aee2a9401: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:15:02.401379 17102 consensus_queue.cc:1035] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:15:02.412137 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:02.428202 30070 consensus_peers.cc:485] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:02.470544 30200 consensus_peers.cc:485] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 02:15:02.486332 483 maintenance_manager.cc:419] P fbbc2be6382e47908ecb7c68c367bd26: Scheduling FlushMRSOp(b6c9fd3cbbd5453897aa072600db569d): perf score=0.033349
I20240503 02:15:02.494927 29872 raft_consensus.cc:1261] T c2d33db64ee8482aad840af219d60b8b P 16440e365430405fb10366ab70e2b429 [term 2 FOLLOWER]: Refusing update from remote peer a9094240860e43bfb129193f3bfd5be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:15:02.495669 16927 consensus_queue.cc:1035] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:15:02.502635 30069 consensus_peers.cc:485] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:02.528462 374 maintenance_manager.cc:643] P fbbc2be6382e47908ecb7c68c367bd26: FlushMRSOp(b6c9fd3cbbd5453897aa072600db569d) complete. Timing: real 0.042s user 0.033s sys 0.005s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":381,"dirs.run_cpu_time_us":199,"dirs.run_wall_time_us":11790,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":731,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":1047,"threads_started":2,"wal-append.queue_time_us":878}
I20240503 02:15:02.530833 483 maintenance_manager.cc:419] P fbbc2be6382e47908ecb7c68c367bd26: Scheduling UndoDeltaBlockGCOp(b6c9fd3cbbd5453897aa072600db569d): 4921 bytes on disk
I20240503 02:15:02.531445 374 maintenance_manager.cc:643] P fbbc2be6382e47908ecb7c68c367bd26: UndoDeltaBlockGCOp(b6c9fd3cbbd5453897aa072600db569d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
W20240503 02:15:02.579953 29809 consensus_peers.cc:485] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:02.597098 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:02.642011 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:02.664312 32709 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:02.683702 30070 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 117 similar messages]
W20240503 02:15:02.684540 30070 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:02.714829 29808 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111) [suppressed 113 similar messages]
W20240503 02:15:02.760543 29940 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 02:15:02.762998 30199 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111) [suppressed 104 similar messages]
W20240503 02:15:02.788295 32709 consensus_peers.cc:485] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:02.798749 30070 consensus_peers.cc:485] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:02.838754 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:02.839821 30070 consensus_peers.cc:485] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:02.872481 32708 consensus_peers.cc:485] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:15:02.915941 29808 consensus_peers.cc:485] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:15:02.951265 371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 84 similar messages]
W20240503 02:15:02.952833 29940 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:02.991864 32708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 02:15:02.992755 32708 consensus_peers.cc:485] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:15:03.085621 32708 consensus_peers.cc:485] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:15:03.105118 30199 consensus_peers.cc:485] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:15:03.115085 30200 consensus_peers.cc:485] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:03.119375 370 consensus_peers.cc:485] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:15:03.132274 30069 consensus_peers.cc:485] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:15:03.147374 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:03.159859 30199 consensus_peers.cc:485] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:15:03.172673 370 consensus_peers.cc:485] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:15:03.180346 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:03.198889 30069 consensus_peers.cc:485] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:15:03.205402 32708 consensus_peers.cc:485] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:15:03.214221 29808 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:15:03.237963 30069 consensus_peers.cc:485] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:15:03.245935 30199 consensus_peers.cc:485] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:15:03.259922 30069 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:15:03.274315 17125 raft_consensus.cc:482] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:03.274479 17125 raft_consensus.cc:504] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:15:03.275104 17125 leader_election.cc:288] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:15:03.276527 29808 leader_election.cc:334] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:03.276697 29809 leader_election.cc:334] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:03.276839 29809 leader_election.cc:302] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 16440e365430405fb10366ab70e2b429; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:03.277087 17125 raft_consensus.cc:2726] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:03.290030 30199 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:15:03.305917 30199 consensus_peers.cc:485] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:15:03.307564 17104 raft_consensus.cc:482] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:03.307756 17104 raft_consensus.cc:504] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:15:03.308375 17104 leader_election.cc:288] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:15:03.309600 370 leader_election.cc:334] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:03.309792 371 leader_election.cc:334] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:03.309911 371 leader_election.cc:302] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:03.310145 17104 raft_consensus.cc:2726] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:03.314182 30199 consensus_peers.cc:485] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:15:03.321548 17128 raft_consensus.cc:482] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:03.321712 17128 raft_consensus.cc:504] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:15:03.322305 17128 leader_election.cc:288] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
W20240503 02:15:03.323750 32709 leader_election.cc:334] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:03.324179 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:03.324505 32708 leader_election.cc:334] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:03.324620 32708 leader_election.cc:302] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:03.324853 17128 raft_consensus.cc:2726] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:15:03.342702 29809 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:03.349640 30199 consensus_peers.cc:485] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:15:03.350158 370 consensus_peers.cc:485] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:15:03.359210 370 consensus_peers.cc:485] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:15:03.379487 30069 consensus_peers.cc:485] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:15:03.387696 29939 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:15:03.423867 32708 consensus_peers.cc:485] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:15:03.425349 32708 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:15:03.434217 371 consensus_peers.cc:485] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:03.451246 29809 consensus_peers.cc:485] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:03.455950 29809 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:03.463620 29808 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:15:03.468191 29808 consensus_peers.cc:485] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:15:03.488921 29808 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:15:03.499025 370 consensus_peers.cc:485] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:15:03.502854 29940 consensus_peers.cc:485] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:03.506229 29809 consensus_peers.cc:485] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:03.543962 30199 consensus_peers.cc:485] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:15:03.551326 370 consensus_peers.cc:485] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:15:03.581954 29808 consensus_peers.cc:485] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:15:03.611667 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:03.674157 16399 raft_consensus.cc:482] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:03.674319 16399 raft_consensus.cc:504] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:15:03.674855 16399 leader_election.cc:288] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:15:03.675947 370 leader_election.cc:334] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:03.676097 371 leader_election.cc:334] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:03.676213 371 leader_election.cc:302] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:03.676422 16399 raft_consensus.cc:2726] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:03.701764 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:03.732736 17131 raft_consensus.cc:482] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:03.732897 17131 raft_consensus.cc:504] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:15:03.733508 17131 leader_election.cc:288] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
W20240503 02:15:03.734717 30070 leader_election.cc:334] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:03.734848 30069 leader_election.cc:334] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:03.734938 30069 leader_election.cc:302] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:03.735157 17131 raft_consensus.cc:2726] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:03.744871 29809 consensus_peers.cc:485] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:03.745836 30199 consensus_peers.cc:485] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:15:03.750701 353 maintenance_manager.cc:419] P 8a666e798fe8477d8dd1e51b385d5ead: Scheduling FlushMRSOp(32a2dbf80daf40d6b21442a39f62f14c): perf score=0.033339
I20240503 02:15:03.787173 32712 maintenance_manager.cc:643] P 8a666e798fe8477d8dd1e51b385d5ead: FlushMRSOp(32a2dbf80daf40d6b21442a39f62f14c) complete. Timing: real 0.036s user 0.034s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":290,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":1318,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":561,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":415,"threads_started":2,"wal-append.queue_time_us":342}
I20240503 02:15:03.789369 353 maintenance_manager.cc:419] P 8a666e798fe8477d8dd1e51b385d5ead: Scheduling UndoDeltaBlockGCOp(32a2dbf80daf40d6b21442a39f62f14c): 4921 bytes on disk
I20240503 02:15:03.789855 32712 maintenance_manager.cc:643] P 8a666e798fe8477d8dd1e51b385d5ead: UndoDeltaBlockGCOp(32a2dbf80daf40d6b21442a39f62f14c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240503 02:15:03.793197 483 maintenance_manager.cc:419] P fbbc2be6382e47908ecb7c68c367bd26: Scheduling FlushMRSOp(df37d88053e2419fbe40d03023d0c756): perf score=0.033371
W20240503 02:15:03.798099 30070 consensus_peers.cc:485] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 02:15:03.848500 374 maintenance_manager.cc:643] P fbbc2be6382e47908ecb7c68c367bd26: FlushMRSOp(df37d88053e2419fbe40d03023d0c756) complete. Timing: real 0.055s user 0.047s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":311,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":1561,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":718,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":1207,"threads_started":2,"wal-append.queue_time_us":1092}
I20240503 02:15:03.850908 483 maintenance_manager.cc:419] P fbbc2be6382e47908ecb7c68c367bd26: Scheduling FlushMRSOp(e61579e74a1a4a9dbd9a09f9b34d9c8d): perf score=0.033340
I20240503 02:15:03.892467 374 maintenance_manager.cc:643] P fbbc2be6382e47908ecb7c68c367bd26: FlushMRSOp(e61579e74a1a4a9dbd9a09f9b34d9c8d) complete. Timing: real 0.041s user 0.019s sys 0.016s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":43,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":1464,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":789,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":222,"threads_started":1,"wal-append.queue_time_us":328}
I20240503 02:15:03.894691 483 maintenance_manager.cc:419] P fbbc2be6382e47908ecb7c68c367bd26: Scheduling UndoDeltaBlockGCOp(df37d88053e2419fbe40d03023d0c756): 4923 bytes on disk
I20240503 02:15:03.895226 374 maintenance_manager.cc:643] P fbbc2be6382e47908ecb7c68c367bd26: UndoDeltaBlockGCOp(df37d88053e2419fbe40d03023d0c756) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240503 02:15:03.896785 483 maintenance_manager.cc:419] P fbbc2be6382e47908ecb7c68c367bd26: Scheduling UndoDeltaBlockGCOp(e61579e74a1a4a9dbd9a09f9b34d9c8d): 4838 bytes on disk
I20240503 02:15:03.897302 374 maintenance_manager.cc:643] P fbbc2be6382e47908ecb7c68c367bd26: UndoDeltaBlockGCOp(e61579e74a1a4a9dbd9a09f9b34d9c8d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
W20240503 02:15:03.904731 30070 consensus_peers.cc:485] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:03.918501 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:03.953416 16927 raft_consensus.cc:482] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:03.953605 16927 raft_consensus.cc:504] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:15:03.954145 16927 leader_election.cc:288] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:15:03.955476 30070 leader_election.cc:334] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:03.955626 30069 leader_election.cc:334] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:03.955722 30069 leader_election.cc:302] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:03.955916 16927 raft_consensus.cc:2726] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:03.957561 32709 consensus_peers.cc:485] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:03.993702 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:04.077447 29809 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:04.083087 30200 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:04.095842 371 consensus_peers.cc:485] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:04.237176 371 consensus_peers.cc:485] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:04.266498 30070 consensus_peers.cc:485] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:04.313728 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:04.350128 371 consensus_peers.cc:485] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:04.396544 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:04.572468 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:04.649262 30070 consensus_peers.cc:485] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:04.661993 29808 consensus_peers.cc:485] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:15:04.668133 29809 consensus_peers.cc:485] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:04.672369 29940 consensus_peers.cc:485] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:04.675963 30200 consensus_peers.cc:485] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:04.756158 29940 consensus_peers.cc:485] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:04.782075 371 consensus_peers.cc:485] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:04.838689 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:04.840108 30069 consensus_peers.cc:485] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:15:04.860800 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:04.862088 30200 consensus_peers.cc:485] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:04.862841 370 consensus_peers.cc:485] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:15:04.916790 30200 consensus_peers.cc:485] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:04.932507 29939 consensus_peers.cc:485] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:15:04.971962 32708 consensus_peers.cc:485] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:15:04.976430 30200 consensus_peers.cc:485] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:04.979337 30070 consensus_peers.cc:485] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:04.985563 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:05.024799 30070 consensus_peers.cc:485] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:05.030386 29809 consensus_peers.cc:485] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:05.031332 30070 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:05.084784 32709 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:05.182997 30070 consensus_peers.cc:485] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:15:05.251348 29808 consensus_peers.cc:485] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:05.254930 32708 consensus_peers.cc:485] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:05.266841 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:05.295411 30069 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:05.295897 29940 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:05.387769 32709 consensus_peers.cc:485] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:05.389343 30070 consensus_peers.cc:485] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:05.464072 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:05.489598 17147 raft_consensus.cc:482] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:05.489774 17147 raft_consensus.cc:504] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:15:05.490509 17147 leader_election.cc:288] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
W20240503 02:15:05.492179 32709 leader_election.cc:334] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:05.494355 32708 leader_election.cc:334] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:05.494450 32708 leader_election.cc:302] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:05.494695 17147 raft_consensus.cc:2726] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:15:05.519497 29808 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:05.549863 32708 consensus_peers.cc:485] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:05.550580 30069 consensus_peers.cc:485] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:05.552003 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:05.558735 370 consensus_peers.cc:485] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:05.570469 32708 consensus_peers.cc:485] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:05.586298 30069 consensus_peers.cc:485] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:05.594120 30200 consensus_peers.cc:485] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:05.608731 370 consensus_peers.cc:485] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:05.611857 30199 consensus_peers.cc:485] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:05.665433 30069 consensus_peers.cc:485] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:05.680806 370 consensus_peers.cc:485] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:05.681056 30069 consensus_peers.cc:485] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:05.684973 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:05.708499 32708 consensus_peers.cc:485] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:05.716074 370 consensus_peers.cc:485] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:05.740682 30199 consensus_peers.cc:485] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:05.744604 30199 consensus_peers.cc:485] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:05.756431 29809 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:05.778391 30199 consensus_peers.cc:485] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:05.788250 32708 consensus_peers.cc:485] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:05.795639 370 consensus_peers.cc:485] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:05.806478 29809 consensus_peers.cc:485] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:05.809700 29809 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:05.819244 29808 consensus_peers.cc:485] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:05.867038 29939 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:05.896665 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:05.898010 32708 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:05.918027 30199 consensus_peers.cc:485] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:15:06.014822 17125 raft_consensus.cc:482] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:06.015030 17125 raft_consensus.cc:504] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:15:06.015681 17125 leader_election.cc:288] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:15:06.016942 29808 leader_election.cc:334] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:06.018086 29809 leader_election.cc:334] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:06.018183 29809 leader_election.cc:302] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 16440e365430405fb10366ab70e2b429; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:06.018389 17125 raft_consensus.cc:2726] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:06.020325 29809 consensus_peers.cc:485] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:06.047787 370 consensus_peers.cc:485] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:06.052021 29940 consensus_peers.cc:485] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:06.072131 30199 consensus_peers.cc:485] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:06.089404 30199 consensus_peers.cc:485] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:06.119871 29808 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:06.121004 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:06.132440 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:06.152000 29808 consensus_peers.cc:485] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:06.190280 30199 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:06.193827 371 consensus_peers.cc:485] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:06.210343 29809 consensus_peers.cc:485] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:06.235610 30199 consensus_peers.cc:485] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:06.292934 29808 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:06.299800 371 consensus_peers.cc:485] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:06.343161 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:06.349314 30070 consensus_peers.cc:485] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:15:06.349578 17131 raft_consensus.cc:482] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:06.349771 17131 raft_consensus.cc:504] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:15:06.350548 17131 leader_election.cc:288] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
W20240503 02:15:06.351917 30069 leader_election.cc:334] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:06.352041 30070 leader_election.cc:334] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:06.352128 30070 leader_election.cc:302] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:06.352339 17131 raft_consensus.cc:2726] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:06.378564 30070 consensus_peers.cc:485] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:06.459338 29809 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 02:15:06.485577 17144 raft_consensus.cc:482] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:06.485786 17144 raft_consensus.cc:504] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:15:06.486472 17144 leader_election.cc:288] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:15:06.488037 370 leader_election.cc:334] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:06.488427 371 leader_election.cc:334] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:06.488513 371 leader_election.cc:302] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:06.488747 17144 raft_consensus.cc:2726] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:15:06.525694 16399 raft_consensus.cc:482] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:06.525883 16399 raft_consensus.cc:504] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:15:06.526588 16399 leader_election.cc:288] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:15:06.528162 370 leader_election.cc:334] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:06.529210 371 leader_election.cc:334] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:06.529364 371 leader_election.cc:302] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:06.529594 17144 raft_consensus.cc:2726] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:06.535202 30200 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:06.563019 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:06.606468 32709 consensus_peers.cc:485] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:06.646227 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:06.673602 371 consensus_peers.cc:485] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:06.677553 371 consensus_peers.cc:485] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:06.822804 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:06.861249 30070 consensus_peers.cc:485] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:07.004411 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:07.103092 29940 consensus_peers.cc:485] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:07.106177 29808 consensus_peers.cc:485] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:07.168186 29809 consensus_peers.cc:485] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:15:07.222820 17131 raft_consensus.cc:482] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:07.223014 17131 raft_consensus.cc:504] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:15:07.223680 17131 leader_election.cc:288] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:15:07.226481 30069 leader_election.cc:334] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:07.226620 30070 leader_election.cc:334] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:07.226697 30069 leader_election.cc:302] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:07.226920 17131 raft_consensus.cc:2726] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:07.230520 30200 consensus_peers.cc:485] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:07.235723 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:07.241652 371 consensus_peers.cc:485] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:07.265542 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:07.290848 30070 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:07.333307 30200 consensus_peers.cc:485] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:15:07.368614 30070 consensus_peers.cc:485] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:07.384954 30069 consensus_peers.cc:485] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:07.394145 30200 consensus_peers.cc:485] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:07.435448 29939 consensus_peers.cc:485] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:07.459947 370 consensus_peers.cc:485] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:07.475860 29940 consensus_peers.cc:485] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:07.480984 30200 consensus_peers.cc:485] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:07.520753 30070 consensus_peers.cc:485] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:07.550170 32708 consensus_peers.cc:485] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:15:07.561362 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:07.586920 30070 consensus_peers.cc:485] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:07.598829 32708 consensus_peers.cc:485] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:15:07.639623 32709 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:07.656229 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:07.662345 30070 consensus_peers.cc:485] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:07.701608 30069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111) [suppressed 158 similar messages]
W20240503 02:15:07.702442 30069 consensus_peers.cc:485] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:15:07.717571 29809 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 154 similar messages]
W20240503 02:15:07.726862 30069 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:15:07.787878 29809 consensus_peers.cc:485] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:07.802325 30199 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111) [suppressed 158 similar messages]
W20240503 02:15:07.807041 32708 consensus_peers.cc:485] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:15:07.816087 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:07.837896 29940 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240503 02:15:07.838864 29940 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:07.847424 370 consensus_peers.cc:485] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:15:07.855343 32709 consensus_peers.cc:485] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:07.872114 29808 consensus_peers.cc:485] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:15:07.903236 30200 consensus_peers.cc:485] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:07.952493 371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240503 02:15:07.957448 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:07.982425 30070 consensus_peers.cc:485] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:15:08.038862 17163 raft_consensus.cc:482] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:08.039079 17163 raft_consensus.cc:504] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:15:08.039808 17163 leader_election.cc:288] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:15:08.041364 371 leader_election.cc:334] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:08.041620 370 leader_election.cc:334] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:08.041746 370 leader_election.cc:302] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:08.042286 17163 raft_consensus.cc:2726] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:08.046869 29808 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:15:08.059283 32709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240503 02:15:08.078351 30199 consensus_peers.cc:485] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:15:08.081768 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:08.151540 17156 raft_consensus.cc:482] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:08.151708 17156 raft_consensus.cc:504] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:15:08.152354 17156 leader_election.cc:288] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:15:08.153605 370 leader_election.cc:334] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:08.154007 371 leader_election.cc:334] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:08.154099 371 leader_election.cc:302] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:08.154291 17156 raft_consensus.cc:2726] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:08.173974 370 consensus_peers.cc:485] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:15:08.198112 30069 consensus_peers.cc:485] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:15:08.217626 30199 consensus_peers.cc:485] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:15:08.217752 30199 consensus_peers.cc:485] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:15:08.223845 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:08.239467 29809 consensus_peers.cc:485] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:08.241091 32708 consensus_peers.cc:485] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:15:08.243192 29809 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:08.259531 30199 consensus_peers.cc:485] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:15:08.271404 29808 consensus_peers.cc:485] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:15:08.274574 32708 consensus_peers.cc:485] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:15:08.277892 29939 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:15:08.288712 30199 consensus_peers.cc:485] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:15:08.307188 17161 raft_consensus.cc:482] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:08.307358 17161 raft_consensus.cc:504] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:15:08.308068 17161 leader_election.cc:288] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
W20240503 02:15:08.309341 32709 leader_election.cc:334] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:08.309464 32708 leader_election.cc:334] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:08.309571 32708 leader_election.cc:302] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:08.309762 17161 raft_consensus.cc:2726] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:15:08.318354 32708 consensus_peers.cc:485] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:15:08.336408 371 consensus_peers.cc:485] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:08.349455 30069 consensus_peers.cc:485] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:15:08.358099 370 consensus_peers.cc:485] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:15:08.411613 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:08.412418 29809 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:08.428942 29940 consensus_peers.cc:485] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:08.445439 32708 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:15:08.446818 30199 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:15:08.459000 30069 consensus_peers.cc:485] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:15:08.467151 370 consensus_peers.cc:485] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:15:08.480624 29809 consensus_peers.cc:485] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:08.558718 370 consensus_peers.cc:485] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:15:08.595608 30199 consensus_peers.cc:485] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:15:08.619192 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:08.638670 30199 consensus_peers.cc:485] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:15:08.658927 29809 consensus_peers.cc:485] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:08.701617 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:08.748585 370 consensus_peers.cc:485] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:15:08.778501 30199 consensus_peers.cc:485] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:15:08.801820 29808 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:15:08.824241 30070 consensus_peers.cc:485] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:08.842567 371 consensus_peers.cc:485] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:08.874756 30070 consensus_peers.cc:485] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:15:08.877673 17151 raft_consensus.cc:482] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:08.877837 17151 raft_consensus.cc:504] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:15:08.878554 17151 leader_election.cc:288] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
W20240503 02:15:08.879776 30070 leader_election.cc:334] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:08.880359 30069 leader_election.cc:334] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:08.880457 30069 leader_election.cc:302] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:08.880674 17151 raft_consensus.cc:2726] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:08.897112 29808 consensus_peers.cc:485] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:15:08.900887 371 consensus_peers.cc:485] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:08.972218 30200 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:08.977212 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:09.010133 29808 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:15:09.024772 29809 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:09.131145 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:09.137950 32709 consensus_peers.cc:485] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:09.221601 371 consensus_peers.cc:485] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:09.252240 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:09.424954 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:09.427968 29809 consensus_peers.cc:485] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:09.574008 30070 consensus_peers.cc:485] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:09.590148 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:09.603606 371 consensus_peers.cc:485] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:09.615319 370 consensus_peers.cc:485] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:15:09.644528 30070 consensus_peers.cc:485] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:09.685945 30200 consensus_peers.cc:485] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:09.731362 29940 consensus_peers.cc:485] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:09.765549 30200 consensus_peers.cc:485] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:09.769263 30200 consensus_peers.cc:485] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:09.783135 29939 consensus_peers.cc:485] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:15:09.814426 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:09.823845 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:09.909133 30070 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:09.930760 29808 consensus_peers.cc:485] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:15:09.959543 30200 consensus_peers.cc:485] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:09.978260 29940 consensus_peers.cc:485] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:10.004830 32708 consensus_peers.cc:485] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:10.008499 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:10.047181 30070 consensus_peers.cc:485] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:15:10.092715 32709 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:10.133824 30070 consensus_peers.cc:485] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:10.157918 30069 consensus_peers.cc:485] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:15:10.173738 30070 consensus_peers.cc:485] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 02:15:10.190711 17157 raft_consensus.cc:482] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:10.190884 17157 raft_consensus.cc:504] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:15:10.191468 17157 leader_election.cc:288] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:15:10.192694 29809 leader_election.cc:334] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:10.192835 29808 leader_election.cc:334] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:10.192958 29808 leader_election.cc:302] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 16440e365430405fb10366ab70e2b429; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:10.193220 17157 raft_consensus.cc:2726] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:10.206521 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:10.224119 30069 consensus_peers.cc:485] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:10.232218 32708 consensus_peers.cc:485] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:15:10.233413 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:10.235033 30199 consensus_peers.cc:485] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:10.259322 30069 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:10.308696 30070 consensus_peers.cc:485] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:10.348484 30200 consensus_peers.cc:485] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:10.366868 32709 consensus_peers.cc:485] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:10.387300 29808 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:10.388034 370 consensus_peers.cc:485] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:10.404203 32708 consensus_peers.cc:485] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:10.426285 29940 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:10.435088 29808 consensus_peers.cc:485] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:10.445684 370 consensus_peers.cc:485] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:10.498294 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:10.525197 17169 raft_consensus.cc:482] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:10.525355 17169 raft_consensus.cc:504] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:15:10.525930 17169 leader_election.cc:288] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:15:10.527458 30069 leader_election.cc:334] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:10.527976 30070 leader_election.cc:334] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:10.528102 30070 leader_election.cc:302] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:10.528353 17169 raft_consensus.cc:2726] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:10.531322 29809 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:10.636279 370 consensus_peers.cc:485] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:10.647737 30199 consensus_peers.cc:485] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:10.647846 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:10.648944 29809 consensus_peers.cc:485] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:10.698544 29808 consensus_peers.cc:485] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:10.704659 32708 consensus_peers.cc:485] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:10.708099 32708 consensus_peers.cc:485] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:10.770063 370 consensus_peers.cc:485] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:10.773605 30199 consensus_peers.cc:485] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:10.786634 30069 consensus_peers.cc:485] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:10.792299 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:10.837772 30199 consensus_peers.cc:485] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:10.871515 30199 consensus_peers.cc:485] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:10.874926 32708 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:10.891489 29809 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:10.904003 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:10.924814 30069 consensus_peers.cc:485] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:10.927295 29939 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:11.002728 32708 consensus_peers.cc:485] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:11.012338 29809 consensus_peers.cc:485] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:11.014454 30069 consensus_peers.cc:485] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:11.025918 30199 consensus_peers.cc:485] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:11.028702 370 consensus_peers.cc:485] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:11.061254 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:11.062266 29940 consensus_peers.cc:485] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:11.079150 370 consensus_peers.cc:485] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:11.093865 29809 consensus_peers.cc:485] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:11.132567 29808 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:11.144057 30199 consensus_peers.cc:485] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:11.174508 30070 consensus_peers.cc:485] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:15:11.201876 30199 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:11.202993 30199 consensus_peers.cc:485] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:11.304541 371 consensus_peers.cc:485] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:11.312170 29808 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:11.326604 29809 consensus_peers.cc:485] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:11.331545 371 consensus_peers.cc:485] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:11.338274 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:11.381212 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:11.413388 29808 consensus_peers.cc:485] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:11.576814 30200 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:11.599973 30070 consensus_peers.cc:485] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:11.602181 29809 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:11.608106 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:11.632812 32709 consensus_peers.cc:485] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:11.739790 371 consensus_peers.cc:485] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:11.750169 30070 consensus_peers.cc:485] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:11.763520 371 consensus_peers.cc:485] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:11.768429 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:11.942323 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:11.950490 30070 consensus_peers.cc:485] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:11.987280 370 consensus_peers.cc:485] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:12.147830 371 consensus_peers.cc:485] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:12.166364 30200 consensus_peers.cc:485] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:12.200186 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:12.203938 30200 consensus_peers.cc:485] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:12.311293 29939 consensus_peers.cc:485] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:12.328176 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:12.333333 29809 consensus_peers.cc:485] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:12.382140 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:12.421221 30200 consensus_peers.cc:485] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:15:12.424799 30200 consensus_peers.cc:485] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:12.430855 29940 consensus_peers.cc:485] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:12.437927 30070 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:12.463992 32708 consensus_peers.cc:485] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:15:12.485109 29808 consensus_peers.cc:485] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:12.490700 30070 consensus_peers.cc:485] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:12.509904 29940 consensus_peers.cc:485] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:12.514014 30070 consensus_peers.cc:485] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:12.638943 30070 consensus_peers.cc:485] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:12.677747 29940 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:12.680070 370 consensus_peers.cc:485] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:15:12.691341 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:12.704927 30070 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 155 similar messages]
W20240503 02:15:12.705750 30070 consensus_peers.cc:485] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:12.720450 29808 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111) [suppressed 148 similar messages]
W20240503 02:15:12.721318 29808 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:15:12.749454 370 consensus_peers.cc:485] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:15:12.760799 30200 consensus_peers.cc:485] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:12.775372 30199 consensus_peers.cc:485] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:15:12.807973 30199 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111) [suppressed 161 similar messages]
W20240503 02:15:12.825680 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:12.841621 30069 consensus_peers.cc:485] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:15:12.841747 30069 consensus_peers.cc:485] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:12.842821 32708 consensus_peers.cc:485] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:15:12.864394 32709 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:12.895181 29939 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240503 02:15:12.908828 30069 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:15:12.910519 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:12.911375 32708 consensus_peers.cc:485] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:15:12.918876 32709 consensus_peers.cc:485] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:12.936111 32708 consensus_peers.cc:485] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:15:12.976188 370 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111) [suppressed 125 similar messages]
W20240503 02:15:12.977003 370 consensus_peers.cc:485] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:15:13.004791 29808 consensus_peers.cc:485] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:15:13.060048 29808 consensus_peers.cc:485] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:15:13.088057 17181 raft_consensus.cc:482] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:13.088217 17181 raft_consensus.cc:504] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:15:13.088773 17181 leader_election.cc:288] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
W20240503 02:15:13.089321 32709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 103 similar messages]
W20240503 02:15:13.090116 32708 leader_election.cc:334] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:13.090240 32709 leader_election.cc:334] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:13.090356 32709 leader_election.cc:302] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:13.090560 17181 raft_consensus.cc:2726] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:15:13.118434 32708 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:15:13.226752 32708 consensus_peers.cc:485] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:15:13.251333 29809 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:13.284215 29809 consensus_peers.cc:485] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:13.302317 29939 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:15:13.314886 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:13.331008 30199 consensus_peers.cc:485] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:15:13.377214 30199 consensus_peers.cc:485] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:15:13.380198 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:13.396927 30199 consensus_peers.cc:485] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:15:13.404134 370 consensus_peers.cc:485] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:15:13.413367 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:13.434068 29809 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:13.434679 30069 consensus_peers.cc:485] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:15:13.438202 32708 consensus_peers.cc:485] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:15:13.519476 30069 consensus_peers.cc:485] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:15:13.530967 29808 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:15:13.563205 29809 consensus_peers.cc:485] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:13.593894 30070 consensus_peers.cc:485] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:13.611407 30199 consensus_peers.cc:485] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:15:13.616633 29940 consensus_peers.cc:485] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:13.618382 30199 consensus_peers.cc:485] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:15:13.621673 17169 raft_consensus.cc:482] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:13.621912 17169 raft_consensus.cc:504] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:15:13.623214 17169 leader_election.cc:288] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
W20240503 02:15:13.624759 30070 leader_election.cc:334] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:13.629176 30199 consensus_peers.cc:485] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:15:13.633983 30069 leader_election.cc:334] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:13.634119 30069 leader_election.cc:302] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:13.634377 17169 raft_consensus.cc:2726] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:13.635247 30069 consensus_peers.cc:485] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:15:13.670794 30199 consensus_peers.cc:485] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:15:13.677449 30199 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:15:13.690272 371 consensus_peers.cc:485] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:13.694466 370 consensus_peers.cc:485] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:15:13.791174 29809 consensus_peers.cc:485] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:13.797482 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:13.798745 30070 consensus_peers.cc:485] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:13.801529 29808 consensus_peers.cc:485] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:15:13.841061 371 consensus_peers.cc:485] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:13.863325 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:13.900677 29808 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:15:13.942997 370 consensus_peers.cc:485] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:15:14.002863 29809 consensus_peers.cc:485] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:14.013397 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:14.018281 29809 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:14.101686 371 consensus_peers.cc:485] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:14.128759 30070 consensus_peers.cc:485] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:14.195041 30200 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:14.207453 371 consensus_peers.cc:485] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 02:15:14.347067 17191 raft_consensus.cc:482] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:14.347236 17191 raft_consensus.cc:504] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:15:14.347915 17191 leader_election.cc:288] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:15:14.349316 370 leader_election.cc:334] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:14.349623 371 leader_election.cc:334] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:14.349732 371 leader_election.cc:302] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:14.349980 17191 raft_consensus.cc:2726] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:14.375380 370 consensus_peers.cc:485] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:15:14.378218 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:14.411267 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:14.451494 30070 consensus_peers.cc:485] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:14.472694 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:14.489130 17169 raft_consensus.cc:482] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:14.489322 17169 raft_consensus.cc:504] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:15:14.490097 17169 leader_election.cc:288] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:15:14.491755 30069 leader_election.cc:334] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:14.492224 32709 consensus_peers.cc:485] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:14.492744 30070 leader_election.cc:334] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:14.492839 30070 leader_election.cc:302] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:14.493072 17169 raft_consensus.cc:2726] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:14.656435 371 consensus_peers.cc:485] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:14.666692 30200 consensus_peers.cc:485] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 02:15:14.708426 17199 raft_consensus.cc:482] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:14.708607 17199 raft_consensus.cc:504] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:15:14.709267 17199 leader_election.cc:288] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:15:14.710482 370 leader_election.cc:334] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:14.710613 371 leader_election.cc:334] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:14.710740 371 leader_election.cc:302] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:14.710913 17199 raft_consensus.cc:2726] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:14.726655 29939 consensus_peers.cc:485] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:15:14.767192 30070 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:14.767702 30200 consensus_peers.cc:485] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:14.791877 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:14.794260 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:14.845103 29940 consensus_peers.cc:485] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:14.854518 32708 consensus_peers.cc:485] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:14.908833 29808 consensus_peers.cc:485] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:15:14.918023 30069 consensus_peers.cc:485] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:14.929572 30200 consensus_peers.cc:485] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:14.972644 29809 consensus_peers.cc:485] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:14.993711 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:15.060168 29940 consensus_peers.cc:485] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:15.063745 30070 consensus_peers.cc:485] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:15.080498 29808 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:15.104236 370 consensus_peers.cc:485] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:15.113675 30199 consensus_peers.cc:485] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:15.131223 30200 consensus_peers.cc:485] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:15.145716 30070 consensus_peers.cc:485] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:15.160986 30070 consensus_peers.cc:485] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:15.220132 29940 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:15.227386 370 consensus_peers.cc:485] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:15.231834 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:15.238166 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:15.246661 32708 consensus_peers.cc:485] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:15:15.300928 32709 consensus_peers.cc:485] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:15.308630 30070 consensus_peers.cc:485] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:15.376513 32708 consensus_peers.cc:485] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:15.389282 32709 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:15.390039 30200 consensus_peers.cc:485] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:15.397606 32708 consensus_peers.cc:485] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:15.468489 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:15.476647 32708 consensus_peers.cc:485] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:15.515523 29808 consensus_peers.cc:485] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:15.530472 30069 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:15.558228 30069 consensus_peers.cc:485] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:15:15.582294 370 consensus_peers.cc:485] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:15.626629 32708 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:15.655437 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:15.699618 29808 consensus_peers.cc:485] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:15.707551 30199 consensus_peers.cc:485] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:15.745374 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:15.745568 32708 consensus_peers.cc:485] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:15.755287 29809 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:15.793690 30199 consensus_peers.cc:485] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:15.846249 30199 consensus_peers.cc:485] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:15.856750 29809 consensus_peers.cc:485] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:15.882939 370 consensus_peers.cc:485] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:15.913218 30199 consensus_peers.cc:485] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:15.922466 29809 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:15.956732 29809 consensus_peers.cc:485] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:15.988680 30069 consensus_peers.cc:485] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:15.989414 30069 consensus_peers.cc:485] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:16.003480 371 consensus_peers.cc:485] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:16.013943 370 consensus_peers.cc:485] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:16.025051 30199 consensus_peers.cc:485] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:16.040241 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:16.042577 29808 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:16.054904 29939 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:16.078410 30069 consensus_peers.cc:485] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:16.180863 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:16.190665 30199 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:16.191402 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:16.230912 29940 consensus_peers.cc:485] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:16.253005 30199 consensus_peers.cc:485] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:16.281729 29809 consensus_peers.cc:485] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:16.326450 30199 consensus_peers.cc:485] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:16.393815 30070 consensus_peers.cc:485] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:16.409783 370 consensus_peers.cc:485] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:16.411266 29808 consensus_peers.cc:485] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:16.447413 371 consensus_peers.cc:485] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:16.449280 29809 consensus_peers.cc:485] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:16.462163 30070 consensus_peers.cc:485] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:16.492007 29809 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:16.501677 371 consensus_peers.cc:485] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:16.561728 371 consensus_peers.cc:485] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:16.594599 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:16.651922 30200 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:16.651996 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:16.660595 30070 consensus_peers.cc:485] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:16.669608 370 consensus_peers.cc:485] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:16.734551 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:16.800585 29808 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:17.060344 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:17.147676 30070 consensus_peers.cc:485] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:17.170948 32709 consensus_peers.cc:485] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:17.172102 30200 consensus_peers.cc:485] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:17.189299 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:17.214524 371 consensus_peers.cc:485] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:17.220038 29939 consensus_peers.cc:485] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:17.228770 30200 consensus_peers.cc:485] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:17.239665 30200 consensus_peers.cc:485] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:17.269708 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:17.279711 32708 consensus_peers.cc:485] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:15:17.325223 17197 raft_consensus.cc:482] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:17.325403 17197 raft_consensus.cc:504] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:15:17.326038 17197 leader_election.cc:288] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:15:17.327353 29808 leader_election.cc:334] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:17.327494 29809 leader_election.cc:334] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:17.327641 29809 leader_election.cc:302] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 16440e365430405fb10366ab70e2b429; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:17.327968 17197 raft_consensus.cc:2726] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:17.344722 30070 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:17.358443 29808 consensus_peers.cc:485] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:17.380901 29940 consensus_peers.cc:485] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:17.387035 30070 consensus_peers.cc:485] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:17.408797 370 consensus_peers.cc:485] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:17.424391 30070 consensus_peers.cc:485] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:17.431849 29940 consensus_peers.cc:485] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:17.444967 32708 consensus_peers.cc:485] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:17.460398 30069 consensus_peers.cc:485] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:17.493865 29808 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:17.538309 29809 consensus_peers.cc:485] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:17.579627 30070 consensus_peers.cc:485] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:17.592133 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:17.597652 30200 consensus_peers.cc:485] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:17.674513 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:17.694298 17219 raft_consensus.cc:482] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:17.694504 17219 raft_consensus.cc:504] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:15:17.695158 17219 leader_election.cc:288] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
W20240503 02:15:17.696647 32709 leader_election.cc:334] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:17.696771 32708 leader_election.cc:334] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:17.696869 32708 leader_election.cc:302] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:17.697101 17219 raft_consensus.cc:2726] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:15:17.699556 29940 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:17.700680 32708 consensus_peers.cc:485] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:17.702896 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:17.706607 30070 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 152 similar messages]
W20240503 02:15:17.723627 29808 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111) [suppressed 148 similar messages]
W20240503 02:15:17.746322 370 consensus_peers.cc:485] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:17.756559 30199 consensus_peers.cc:485] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:17.760802 32709 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:17.763239 32708 consensus_peers.cc:485] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:17.829690 30199 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111) [suppressed 159 similar messages]
W20240503 02:15:17.894567 32708 consensus_peers.cc:485] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:17.899276 30200 consensus_peers.cc:485] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:17.929922 32709 consensus_peers.cc:485] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:17.969564 29940 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240503 02:15:17.970912 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:18.060761 370 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111) [suppressed 124 similar messages]
W20240503 02:15:18.071691 32708 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:18.074731 30070 consensus_peers.cc:485] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:18.106918 370 consensus_peers.cc:485] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:18.121737 30199 consensus_peers.cc:485] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:18.127846 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:18.155184 32709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 105 similar messages]
W20240503 02:15:18.159417 29808 consensus_peers.cc:485] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:18.196702 30069 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:18.199302 30069 consensus_peers.cc:485] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:15:18.201273 32708 consensus_peers.cc:485] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:18.215893 29808 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:18.219600 30199 consensus_peers.cc:485] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:18.243865 30069 consensus_peers.cc:485] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:18.250571 29808 consensus_peers.cc:485] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:18.271131 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:18.288038 30199 consensus_peers.cc:485] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:18.289590 29809 consensus_peers.cc:485] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:18.302688 370 consensus_peers.cc:485] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:18.355674 30199 consensus_peers.cc:485] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:18.359454 371 consensus_peers.cc:485] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:18.361690 30199 consensus_peers.cc:485] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:18.389724 29809 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:18.423866 30069 consensus_peers.cc:485] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:18.437649 29809 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:18.455965 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:18.548094 29939 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:18.562419 29809 consensus_peers.cc:485] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:18.610862 370 consensus_peers.cc:485] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:18.644217 29940 consensus_peers.cc:485] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:18.678260 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:18.691354 30070 consensus_peers.cc:485] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:18.719331 30199 consensus_peers.cc:485] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:18.722709 30069 consensus_peers.cc:485] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:18.725064 30070 consensus_peers.cc:485] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:18.821774 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:18.821861 29809 consensus_peers.cc:485] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:18.897857 30199 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:18.983479 30199 consensus_peers.cc:485] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:18.996320 29809 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:19.005115 371 consensus_peers.cc:485] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:19.022758 29809 consensus_peers.cc:485] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 02:15:19.023945 17222 raft_consensus.cc:482] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:19.024137 17222 raft_consensus.cc:504] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:15:19.024739 17222 leader_election.cc:288] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:15:19.026314 370 leader_election.cc:334] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:19.026721 371 leader_election.cc:334] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:19.026825 371 leader_election.cc:302] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:19.027061 17222 raft_consensus.cc:2726] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:19.029433 371 consensus_peers.cc:485] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:19.046478 29808 consensus_peers.cc:485] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:19.069651 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:19.074507 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:19.108994 371 consensus_peers.cc:485] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:19.110886 370 consensus_peers.cc:485] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:19.186327 30070 consensus_peers.cc:485] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:19.211143 370 consensus_peers.cc:485] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:19.215142 30200 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:19.226542 29808 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:19.402621 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:19.494171 32709 consensus_peers.cc:485] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:19.598248 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:19.601382 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:19.636981 371 consensus_peers.cc:485] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:19.650161 29939 consensus_peers.cc:485] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:19.680675 29808 consensus_peers.cc:485] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:19.685137 30200 consensus_peers.cc:485] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:19.685297 30200 consensus_peers.cc:485] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:19.769719 32708 consensus_peers.cc:485] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:19.778046 29940 consensus_peers.cc:485] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:19.791190 370 consensus_peers.cc:485] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:19.794925 29940 consensus_peers.cc:485] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:19.796208 30069 consensus_peers.cc:485] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:19.798079 30070 consensus_peers.cc:485] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:19.839227 30070 consensus_peers.cc:485] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:19.863315 32708 consensus_peers.cc:485] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:19.918944 30200 consensus_peers.cc:485] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:19.923132 29808 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:19.924557 30070 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:19.978668 30070 consensus_peers.cc:485] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:19.984699 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:19.993245 30070 consensus_peers.cc:485] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:20.062880 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:20.107581 29940 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:20.145596 29809 consensus_peers.cc:485] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:20.145681 30200 consensus_peers.cc:485] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:20.166728 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:20.207979 30199 consensus_peers.cc:485] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:20.216894 32709 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:20.219115 32708 consensus_peers.cc:485] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:20.323251 30200 consensus_peers.cc:485] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:20.341637 32708 consensus_peers.cc:485] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:20.364955 32708 consensus_peers.cc:485] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:20.472882 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:20.476627 32709 consensus_peers.cc:485] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:20.493572 370 consensus_peers.cc:485] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:20.517846 29809 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:20.521210 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:20.536231 30069 consensus_peers.cc:485] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:15:20.539260 30069 consensus_peers.cc:485] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:20.540475 32708 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:20.593801 370 consensus_peers.cc:485] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:20.609946 29808 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:20.629182 30069 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:20.685086 30069 consensus_peers.cc:485] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:20.712584 30199 consensus_peers.cc:485] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:20.716428 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:20.742203 30070 consensus_peers.cc:485] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:20.743228 29808 consensus_peers.cc:485] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:20.751089 32708 consensus_peers.cc:485] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:20.777242 29808 consensus_peers.cc:485] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:20.781590 30199 consensus_peers.cc:485] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:20.783135 371 consensus_peers.cc:485] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:20.820327 30199 consensus_peers.cc:485] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:20.857401 30199 consensus_peers.cc:485] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:20.873448 29809 consensus_peers.cc:485] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:20.879992 29809 consensus_peers.cc:485] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:20.950142 370 consensus_peers.cc:485] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:20.952144 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:21.047298 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:21.107947 370 consensus_peers.cc:485] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:21.117259 29940 consensus_peers.cc:485] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:21.118479 29939 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:21.130867 30069 consensus_peers.cc:485] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:21.144147 29809 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:21.179929 30070 consensus_peers.cc:485] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:21.230300 30199 consensus_peers.cc:485] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:21.238221 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:21.260936 30070 consensus_peers.cc:485] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:21.277204 29809 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:21.288435 29809 consensus_peers.cc:485] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:21.324261 30199 consensus_peers.cc:485] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:21.324430 30199 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:21.353472 371 consensus_peers.cc:485] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:21.381660 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:21.439795 29808 consensus_peers.cc:485] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:21.464701 30200 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:21.467691 29809 consensus_peers.cc:485] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:21.474697 370 consensus_peers.cc:485] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:21.487875 30199 consensus_peers.cc:485] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:21.547955 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:21.622690 371 consensus_peers.cc:485] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:21.653535 371 consensus_peers.cc:485] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 02:15:21.738140 17222 raft_consensus.cc:482] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:21.738312 17222 raft_consensus.cc:504] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:15:21.738932 17222 leader_election.cc:288] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:15:21.740267 370 leader_election.cc:334] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:21.740458 371 leader_election.cc:334] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:21.740574 371 leader_election.cc:302] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:21.740803 17222 raft_consensus.cc:2726] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:21.842056 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:21.888422 370 consensus_peers.cc:485] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:21.964454 29808 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:21.972402 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:21.977581 30070 consensus_peers.cc:485] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:22.018657 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:22.071974 17232 raft_consensus.cc:482] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:22.072264 17232 raft_consensus.cc:504] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:15:22.073067 17232 leader_election.cc:288] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
W20240503 02:15:22.074230 30070 leader_election.cc:334] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:22.075735 30069 leader_election.cc:334] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:22.075852 30069 leader_election.cc:302] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:22.076117 17232 raft_consensus.cc:2726] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:22.088408 32708 consensus_peers.cc:485] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:22.106009 29808 consensus_peers.cc:485] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:22.108206 30200 consensus_peers.cc:485] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:22.122296 30070 consensus_peers.cc:485] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:22.135628 32709 consensus_peers.cc:485] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:22.141831 30069 consensus_peers.cc:485] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:22.198016 30200 consensus_peers.cc:485] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:22.221978 370 consensus_peers.cc:485] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:22.229219 371 consensus_peers.cc:485] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:22.230880 32708 consensus_peers.cc:485] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:22.301398 29940 consensus_peers.cc:485] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:22.317793 30070 consensus_peers.cc:485] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:22.325817 29939 consensus_peers.cc:485] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:22.358318 29940 consensus_peers.cc:485] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:22.364986 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:22.402333 30200 consensus_peers.cc:485] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:22.412652 30070 consensus_peers.cc:485] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:22.418215 30070 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:22.424798 30200 consensus_peers.cc:485] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:22.449579 30070 consensus_peers.cc:485] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:22.480939 29940 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 02:15:22.494127 17227 raft_consensus.cc:482] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:22.494295 17227 raft_consensus.cc:504] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:15:22.494987 17227 leader_election.cc:288] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:15:22.496558 29809 leader_election.cc:334] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:22.496731 29808 leader_election.cc:334] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:22.496858 29808 leader_election.cc:302] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 16440e365430405fb10366ab70e2b429; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:22.497117 17227 raft_consensus.cc:2726] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:22.557211 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:22.566843 29808 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:22.608459 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:22.642560 32708 consensus_peers.cc:485] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:22.698653 29809 consensus_peers.cc:485] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:22.712352 30070 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 154 similar messages]
W20240503 02:15:22.725112 32708 consensus_peers.cc:485] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:22.737219 32709 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:22.744680 29809 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 154 similar messages]
W20240503 02:15:22.849066 30199 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111) [suppressed 154 similar messages]
W20240503 02:15:22.854926 30199 consensus_peers.cc:485] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:22.872826 370 consensus_peers.cc:485] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:22.883764 30200 consensus_peers.cc:485] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:22.894745 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:22.900561 30069 consensus_peers.cc:485] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:22.960729 30069 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:22.967211 29808 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:23.030836 29940 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240503 02:15:23.031195 30199 consensus_peers.cc:485] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:23.031692 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:23.069062 32709 consensus_peers.cc:485] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:23.074056 370 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111) [suppressed 126 similar messages]
W20240503 02:15:23.074911 370 consensus_peers.cc:485] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:23.080792 32708 consensus_peers.cc:485] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:23.108647 32708 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:23.118636 30069 consensus_peers.cc:485] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:23.122812 29809 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:23.139483 29808 consensus_peers.cc:485] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:23.169812 30069 consensus_peers.cc:485] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:15:23.178172 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:23.181398 32708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240503 02:15:23.192319 29809 consensus_peers.cc:485] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:23.214555 29808 consensus_peers.cc:485] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:23.255982 29940 consensus_peers.cc:485] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:23.290616 371 consensus_peers.cc:485] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:23.347836 30199 consensus_peers.cc:485] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:23.378832 32708 consensus_peers.cc:485] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:23.385316 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:23.429342 370 consensus_peers.cc:485] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:23.448563 30199 consensus_peers.cc:485] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:23.458806 30199 consensus_peers.cc:485] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:23.464223 30070 consensus_peers.cc:485] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:23.480756 29809 consensus_peers.cc:485] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:23.506564 370 consensus_peers.cc:485] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:23.595634 30070 consensus_peers.cc:485] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:23.621544 29939 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:23.643682 370 consensus_peers.cc:485] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:23.648339 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:23.648550 17228 raft_consensus.cc:482] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:23.648706 17228 raft_consensus.cc:504] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:15:23.649410 17228 leader_election.cc:288] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:15:23.650887 30070 leader_election.cc:334] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:23.651124 30069 leader_election.cc:334] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:23.651240 30069 leader_election.cc:302] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:23.651424 17228 raft_consensus.cc:2726] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:23.669903 30069 consensus_peers.cc:485] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:23.720692 30199 consensus_peers.cc:485] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:23.742601 30199 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:23.755023 30199 consensus_peers.cc:485] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:23.764611 371 consensus_peers.cc:485] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:23.771173 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:23.774384 30070 consensus_peers.cc:485] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:23.793372 29809 consensus_peers.cc:485] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:23.808599 29809 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:23.859077 29809 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:23.879817 29809 consensus_peers.cc:485] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:23.954414 30199 consensus_peers.cc:485] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:23.962013 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:24.084435 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:24.116394 371 consensus_peers.cc:485] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:24.126016 30200 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:24.136143 29808 consensus_peers.cc:485] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:24.153000 371 consensus_peers.cc:485] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:24.176795 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:24.377516 32708 consensus_peers.cc:485] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:24.402819 29808 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:24.468050 30070 consensus_peers.cc:485] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:24.516072 370 consensus_peers.cc:485] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:24.527670 32709 consensus_peers.cc:485] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:24.528394 30069 consensus_peers.cc:485] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:24.551196 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:24.569609 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:24.622781 29939 consensus_peers.cc:485] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:24.637307 30200 consensus_peers.cc:485] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:24.652164 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:24.660600 30070 consensus_peers.cc:485] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:24.671187 370 consensus_peers.cc:485] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:24.694010 30200 consensus_peers.cc:485] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:24.795120 32708 consensus_peers.cc:485] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:24.808063 29940 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:24.819088 29808 consensus_peers.cc:485] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:24.829277 30070 consensus_peers.cc:485] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:24.848340 371 consensus_peers.cc:485] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:24.866756 30200 consensus_peers.cc:485] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:24.875973 32708 consensus_peers.cc:485] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:24.898034 30070 consensus_peers.cc:485] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:24.919396 30200 consensus_peers.cc:485] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:24.940594 29940 consensus_peers.cc:485] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:24.985399 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:25.007035 29940 consensus_peers.cc:485] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:25.007046 30070 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:25.030064 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:25.049572 30070 consensus_peers.cc:485] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:25.140131 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:25.153018 32708 consensus_peers.cc:485] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:25.184224 30069 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:25.258222 32709 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:25.265110 29809 consensus_peers.cc:485] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:25.288424 30199 consensus_peers.cc:485] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:25.305817 370 consensus_peers.cc:485] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:25.335060 29808 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:25.335350 30069 consensus_peers.cc:485] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:25.442411 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:25.485420 30200 consensus_peers.cc:485] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:25.518499 30199 consensus_peers.cc:485] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:25.539891 30069 consensus_peers.cc:485] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:15:25.555028 29808 consensus_peers.cc:485] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:25.580559 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:25.584926 29809 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:25.608525 30069 consensus_peers.cc:485] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:25.621995 32709 consensus_peers.cc:485] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:25.651461 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:25.697955 32708 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:25.701463 29808 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:25.705325 30199 consensus_peers.cc:485] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:25.708309 30199 consensus_peers.cc:485] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:25.756230 370 consensus_peers.cc:485] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:25.803261 30070 consensus_peers.cc:485] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:25.811455 29809 consensus_peers.cc:485] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:25.862123 29809 consensus_peers.cc:485] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:25.916148 30199 consensus_peers.cc:485] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:25.918485 371 consensus_peers.cc:485] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:25.920348 29808 consensus_peers.cc:485] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:25.925585 370 consensus_peers.cc:485] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:25.935643 29940 consensus_peers.cc:485] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:25.948241 32708 consensus_peers.cc:485] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:25.954084 32708 consensus_peers.cc:485] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:25.954104 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:25.996148 370 consensus_peers.cc:485] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:26.046022 30199 consensus_peers.cc:485] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:26.060101 30070 consensus_peers.cc:485] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:26.092356 29939 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:26.111810 370 consensus_peers.cc:485] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 02:15:26.137746 17248 raft_consensus.cc:482] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:26.137940 17248 raft_consensus.cc:504] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:15:26.138636 17248 leader_election.cc:288] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:15:26.139937 370 leader_election.cc:334] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:26.140080 371 leader_election.cc:334] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:26.140187 371 leader_election.cc:302] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:26.140434 17248 raft_consensus.cc:2726] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:26.173678 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:26.178946 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:26.190369 17254 raft_consensus.cc:482] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:26.190498 17254 raft_consensus.cc:504] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:15:26.191064 17254 leader_election.cc:288] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
W20240503 02:15:26.192304 32709 leader_election.cc:334] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:26.192440 32708 leader_election.cc:334] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:26.192548 32708 leader_election.cc:302] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:26.192734 17254 raft_consensus.cc:2726] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:15:26.230109 29809 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:26.257297 29809 consensus_peers.cc:485] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:26.302683 371 consensus_peers.cc:485] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:26.315196 30069 consensus_peers.cc:485] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:26.322252 30070 consensus_peers.cc:485] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:26.327153 29809 consensus_peers.cc:485] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:26.386647 30199 consensus_peers.cc:485] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:26.389014 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:26.399974 30199 consensus_peers.cc:485] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:26.455852 30199 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:26.478036 29808 consensus_peers.cc:485] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:26.520366 29809 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:26.596783 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:26.614104 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:26.698472 371 consensus_peers.cc:485] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:26.804834 371 consensus_peers.cc:485] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:26.843312 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:26.875236 32708 consensus_peers.cc:485] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:26.903667 370 consensus_peers.cc:485] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:26.911587 29808 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:26.943061 32709 consensus_peers.cc:485] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:26.954741 30070 consensus_peers.cc:485] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:26.993729 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:27.024163 30200 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:27.024179 30070 consensus_peers.cc:485] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:27.048463 370 consensus_peers.cc:485] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:27.070899 30200 consensus_peers.cc:485] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:27.095934 30069 consensus_peers.cc:485] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:27.141269 30200 consensus_peers.cc:485] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:27.147634 29808 consensus_peers.cc:485] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:27.194731 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:27.206679 32708 consensus_peers.cc:485] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:27.229750 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:27.232928 29940 consensus_peers.cc:485] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:27.265822 30070 consensus_peers.cc:485] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:27.271957 30200 consensus_peers.cc:485] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:27.272102 30200 consensus_peers.cc:485] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:27.273519 30070 consensus_peers.cc:485] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:27.302623 30070 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:27.336791 29940 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:27.369796 30070 consensus_peers.cc:485] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:27.377741 29939 consensus_peers.cc:485] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:27.453588 371 consensus_peers.cc:485] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:27.544687 32708 consensus_peers.cc:485] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:27.577458 29940 consensus_peers.cc:485] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:27.608259 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:27.636955 29809 consensus_peers.cc:485] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:27.648130 30199 consensus_peers.cc:485] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:27.688037 30199 consensus_peers.cc:485] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:27.730458 30070 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 150 similar messages]
W20240503 02:15:27.735044 32709 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:27.740622 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:27.746901 370 consensus_peers.cc:485] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:27.755514 29809 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 147 similar messages]
W20240503 02:15:27.774315 32708 consensus_peers.cc:485] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:27.798183 30069 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:27.804409 30069 consensus_peers.cc:485] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:27.847352 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:27.854856 30069 consensus_peers.cc:485] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:27.903942 29808 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:27.912057 30199 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111) [suppressed 164 similar messages]
W20240503 02:15:28.010455 32709 consensus_peers.cc:485] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:28.026733 30200 consensus_peers.cc:485] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:28.130227 29808 consensus_peers.cc:485] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:28.131403 29940 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240503 02:15:28.173274 30199 consensus_peers.cc:485] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:28.177265 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:28.197443 30070 consensus_peers.cc:485] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:28.224025 370 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240503 02:15:28.252125 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:28.257381 32708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240503 02:15:28.258157 32708 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:28.261195 29809 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:28.274282 370 consensus_peers.cc:485] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:28.286993 29808 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:28.295930 30199 consensus_peers.cc:485] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:28.317982 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:28.335131 370 consensus_peers.cc:485] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:28.348273 32708 consensus_peers.cc:485] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:28.418148 370 consensus_peers.cc:485] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:28.426210 29808 consensus_peers.cc:485] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:28.429580 29809 consensus_peers.cc:485] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:28.446518 30069 consensus_peers.cc:485] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:15:28.465839 32708 consensus_peers.cc:485] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:28.484282 371 consensus_peers.cc:485] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:28.521651 29809 consensus_peers.cc:485] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:28.558712 30199 consensus_peers.cc:485] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:28.630416 29809 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:28.631523 30070 consensus_peers.cc:485] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:28.668965 29940 consensus_peers.cc:485] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:28.699311 30070 consensus_peers.cc:485] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:28.727793 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:28.735483 370 consensus_peers.cc:485] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:28.774179 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:28.793900 30199 consensus_peers.cc:485] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:28.806242 29809 consensus_peers.cc:485] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:28.820626 30069 consensus_peers.cc:485] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:28.833403 29939 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:28.855118 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:28.885432 29809 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:28.896443 30199 consensus_peers.cc:485] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:28.943226 29809 consensus_peers.cc:485] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:28.962023 371 consensus_peers.cc:485] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:29.022379 371 consensus_peers.cc:485] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:29.094377 30199 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:29.188783 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:29.197531 29808 consensus_peers.cc:485] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:29.232414 30199 consensus_peers.cc:485] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:29.270742 30070 consensus_peers.cc:485] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:29.289260 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:29.310055 32708 consensus_peers.cc:485] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:29.342676 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:29.343740 370 consensus_peers.cc:485] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:29.354274 371 consensus_peers.cc:485] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:29.386193 29808 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:29.499011 30200 consensus_peers.cc:485] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:29.500946 30200 consensus_peers.cc:485] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:29.508536 370 consensus_peers.cc:485] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:29.532215 30200 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:29.547457 30070 consensus_peers.cc:485] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 02:15:29.583750 483 maintenance_manager.cc:419] P fbbc2be6382e47908ecb7c68c367bd26: Scheduling FlushMRSOp(63b416f4e004490b8e38a96fd065dfea): perf score=0.033345
W20240503 02:15:29.629451 30069 consensus_peers.cc:485] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:15:29.640101 374 maintenance_manager.cc:643] P fbbc2be6382e47908ecb7c68c367bd26: FlushMRSOp(63b416f4e004490b8e38a96fd065dfea) complete. Timing: real 0.056s user 0.035s sys 0.019s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":330,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":986,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":665,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":564,"threads_started":2,"wal-append.queue_time_us":484}
I20240503 02:15:29.642670 483 maintenance_manager.cc:419] P fbbc2be6382e47908ecb7c68c367bd26: Scheduling UndoDeltaBlockGCOp(63b416f4e004490b8e38a96fd065dfea): 4923 bytes on disk
I20240503 02:15:29.643229 374 maintenance_manager.cc:643] P fbbc2be6382e47908ecb7c68c367bd26: UndoDeltaBlockGCOp(63b416f4e004490b8e38a96fd065dfea) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
W20240503 02:15:29.645325 32709 consensus_peers.cc:485] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:29.650458 29808 consensus_peers.cc:485] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:29.698299 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:29.701287 30200 consensus_peers.cc:485] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:29.734428 29940 consensus_peers.cc:485] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:29.742947 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:29.774238 29939 consensus_peers.cc:485] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:29.782177 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:29.853689 30070 consensus_peers.cc:485] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:29.879254 30070 consensus_peers.cc:485] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:29.880244 29809 consensus_peers.cc:485] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:29.891948 371 consensus_peers.cc:485] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:29.893749 30070 consensus_peers.cc:485] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:29.934625 32708 consensus_peers.cc:485] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:29.934959 30070 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:29.967366 29940 consensus_peers.cc:485] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:29.991360 30200 consensus_peers.cc:485] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 02:15:29.994848 17277 raft_consensus.cc:482] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:29.995050 17277 raft_consensus.cc:504] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:15:29.995719 17277 leader_election.cc:288] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:15:29.996971 30069 leader_election.cc:334] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:29.997176 30070 leader_election.cc:334] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:29.997301 30070 leader_election.cc:302] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:29.997547 17277 raft_consensus.cc:2726] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:30.018813 32708 consensus_peers.cc:485] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:30.110651 29940 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:30.113694 30199 consensus_peers.cc:485] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:15:30.149009 483 maintenance_manager.cc:419] P fbbc2be6382e47908ecb7c68c367bd26: Scheduling FlushMRSOp(04e47da8cf7445dcacaf1f7c8950bd34): perf score=0.033366
I20240503 02:15:30.185693 374 maintenance_manager.cc:643] P fbbc2be6382e47908ecb7c68c367bd26: FlushMRSOp(04e47da8cf7445dcacaf1f7c8950bd34) complete. Timing: real 0.036s user 0.019s sys 0.016s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":400,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":1821,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":627,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":1499,"threads_started":2,"wal-append.queue_time_us":1893}
I20240503 02:15:30.188369 483 maintenance_manager.cc:419] P fbbc2be6382e47908ecb7c68c367bd26: Scheduling UndoDeltaBlockGCOp(04e47da8cf7445dcacaf1f7c8950bd34): 4836 bytes on disk
I20240503 02:15:30.189288 374 maintenance_manager.cc:643] P fbbc2be6382e47908ecb7c68c367bd26: UndoDeltaBlockGCOp(04e47da8cf7445dcacaf1f7c8950bd34) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":167,"lbm_reads_lt_1ms":4}
W20240503 02:15:30.201453 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:30.202919 30199 consensus_peers.cc:485] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:30.233878 32709 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:30.280520 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:30.350314 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:30.358652 29808 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:30.397290 370 consensus_peers.cc:485] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:30.420053 30200 consensus_peers.cc:485] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:30.422616 32709 consensus_peers.cc:485] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:30.446938 32708 consensus_peers.cc:485] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:30.448189 30069 consensus_peers.cc:485] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:30.467581 30069 consensus_peers.cc:485] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:30.556958 29808 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:30.560295 29808 consensus_peers.cc:485] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:30.658664 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:30.681017 30069 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:30.724059 29809 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:30.734705 370 consensus_peers.cc:485] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:30.761019 32708 consensus_peers.cc:485] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:30.764729 32708 consensus_peers.cc:485] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:30.773051 30199 consensus_peers.cc:485] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:30.826416 29809 consensus_peers.cc:485] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:30.836251 32708 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:30.842803 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:30.858561 371 consensus_peers.cc:485] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:30.862341 370 consensus_peers.cc:485] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:30.889314 29809 consensus_peers.cc:485] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:30.890848 370 consensus_peers.cc:485] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:30.904356 30199 consensus_peers.cc:485] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:30.943688 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:30.948627 30199 consensus_peers.cc:485] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:30.952492 30070 consensus_peers.cc:485] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:30.978886 29808 consensus_peers.cc:485] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:31.038200 30069 consensus_peers.cc:485] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:15:31.042188 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:31.089494 29940 consensus_peers.cc:485] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:31.094261 29809 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:31.104144 30199 consensus_peers.cc:485] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:31.202966 30069 consensus_peers.cc:485] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:31.210912 30070 consensus_peers.cc:485] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:31.287837 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:31.294687 29939 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:31.295482 30070 consensus_peers.cc:485] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:31.359099 370 consensus_peers.cc:485] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:31.389387 371 consensus_peers.cc:485] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:31.389498 29809 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:31.406845 29809 consensus_peers.cc:485] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:31.491051 30199 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:31.499213 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:31.540593 30199 consensus_peers.cc:485] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:31.563686 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:31.579823 30199 consensus_peers.cc:485] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:31.583935 30070 consensus_peers.cc:485] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:31.672447 371 consensus_peers.cc:485] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 02:15:31.678305 17280 raft_consensus.cc:482] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
W20240503 02:15:31.678426 29809 consensus_peers.cc:485] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 02:15:31.678462 17280 raft_consensus.cc:504] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:15:31.679180 17280 leader_election.cc:288] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:15:31.680604 29808 leader_election.cc:334] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:31.680725 29809 leader_election.cc:334] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:31.680827 29809 leader_election.cc:302] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 16440e365430405fb10366ab70e2b429; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:31.681072 17280 raft_consensus.cc:2726] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:31.727345 29808 consensus_peers.cc:485] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:31.748718 30200 consensus_peers.cc:485] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:31.778441 30069 consensus_peers.cc:485] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:31.809489 29808 consensus_peers.cc:485] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:31.815176 30070 consensus_peers.cc:485] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:31.819517 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:31.827522 370 consensus_peers.cc:485] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:31.894268 29808 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:31.957551 32708 consensus_peers.cc:485] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:31.970705 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:31.988555 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:32.017058 30200 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:32.101055 29939 consensus_peers.cc:485] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:32.146643 371 consensus_peers.cc:485] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:32.151129 32709 consensus_peers.cc:485] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:32.250702 30200 consensus_peers.cc:485] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:32.253245 30200 consensus_peers.cc:485] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:32.267132 370 consensus_peers.cc:485] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:32.322366 30070 consensus_peers.cc:485] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:32.332727 29940 consensus_peers.cc:485] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:32.360388 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:32.362212 29940 consensus_peers.cc:485] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:32.399830 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:32.460377 371 consensus_peers.cc:485] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:32.505686 29809 consensus_peers.cc:485] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:32.521111 30070 consensus_peers.cc:485] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:32.521577 32708 consensus_peers.cc:485] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:32.539626 32708 consensus_peers.cc:485] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:32.546186 32709 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:32.557873 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:32.562855 17292 raft_consensus.cc:482] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:32.563015 17292 raft_consensus.cc:504] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:15:32.563664 17292 leader_election.cc:288] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
W20240503 02:15:32.564888 30070 leader_election.cc:334] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:32.565086 30069 leader_election.cc:334] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:32.565209 30069 leader_election.cc:302] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:32.565405 17292 raft_consensus.cc:2726] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:32.599872 30200 consensus_peers.cc:485] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:32.604987 30070 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:32.658491 32709 consensus_peers.cc:485] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:32.677616 30199 consensus_peers.cc:485] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:32.691659 29940 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:32.699797 30070 consensus_peers.cc:485] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:32.763643 29808 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111) [suppressed 153 similar messages]
W20240503 02:15:32.765502 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:32.765519 30070 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 153 similar messages]
W20240503 02:15:32.768141 30199 consensus_peers.cc:485] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:32.791332 29808 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:32.878700 370 consensus_peers.cc:485] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:32.924589 30199 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111) [suppressed 157 similar messages]
W20240503 02:15:32.930815 32708 consensus_peers.cc:485] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:32.957607 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:32.983335 30069 consensus_peers.cc:485] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:33.017575 30069 consensus_peers.cc:485] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:33.021916 29809 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:33.049829 30200 consensus_peers.cc:485] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:33.077391 30069 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:33.099920 29808 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:33.132902 29808 consensus_peers.cc:485] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:33.158077 29940 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240503 02:15:33.158885 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:33.203498 370 consensus_peers.cc:485] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:33.240579 371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240503 02:15:33.258244 30199 consensus_peers.cc:485] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:33.294206 30070 consensus_peers.cc:485] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:33.321878 371 consensus_peers.cc:485] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:33.322091 32708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240503 02:15:33.323877 32708 consensus_peers.cc:485] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:33.339525 29809 consensus_peers.cc:485] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:33.344656 30199 consensus_peers.cc:485] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:33.379899 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:33.399791 32708 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:33.405649 32708 consensus_peers.cc:485] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:33.409303 370 consensus_peers.cc:485] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:33.423460 370 consensus_peers.cc:485] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:33.439352 29940 consensus_peers.cc:485] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:33.491173 29809 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:33.506839 30199 consensus_peers.cc:485] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:33.547040 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:33.561851 30199 consensus_peers.cc:485] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:33.603415 30069 consensus_peers.cc:485] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:15:33.637904 29809 consensus_peers.cc:485] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:33.640911 29808 consensus_peers.cc:485] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:33.711325 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:33.740948 30069 consensus_peers.cc:485] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 02:15:33.765348 17294 raft_consensus.cc:482] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:33.765551 17294 raft_consensus.cc:504] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:15:33.766775 17294 leader_election.cc:288] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:15:33.768121 370 leader_election.cc:334] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:33.768252 371 leader_election.cc:334] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:33.768363 371 leader_election.cc:302] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:33.768563 17294 raft_consensus.cc:2726] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:33.770985 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:33.773650 29809 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:33.779246 29939 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:33.813949 370 consensus_peers.cc:485] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:33.815515 30070 consensus_peers.cc:485] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:34.003909 370 consensus_peers.cc:485] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:34.061604 30070 consensus_peers.cc:485] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:34.061753 29809 consensus_peers.cc:485] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:34.070802 30199 consensus_peers.cc:485] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:34.079412 371 consensus_peers.cc:485] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:34.082388 29809 consensus_peers.cc:485] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:34.093647 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:34.141081 30199 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:34.163688 30199 consensus_peers.cc:485] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:34.186084 30070 consensus_peers.cc:485] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:34.188136 371 consensus_peers.cc:485] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:34.196388 29808 consensus_peers.cc:485] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:34.218719 30200 consensus_peers.cc:485] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:34.264603 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:34.264691 29808 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 02:15:34.279246 17301 raft_consensus.cc:482] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:34.279428 17301 raft_consensus.cc:504] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:15:34.280133 17301 leader_election.cc:288] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
W20240503 02:15:34.281574 32709 leader_election.cc:334] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:34.281960 32708 leader_election.cc:334] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:34.282053 32708 leader_election.cc:302] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:34.282289 17268 raft_consensus.cc:2726] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:15:34.335037 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:34.415000 30070 consensus_peers.cc:485] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:34.431591 30200 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:34.450469 30069 consensus_peers.cc:485] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:34.472445 29808 consensus_peers.cc:485] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:34.550539 30070 consensus_peers.cc:485] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:34.590276 29940 consensus_peers.cc:485] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:34.591105 30200 consensus_peers.cc:485] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:34.620909 32709 consensus_peers.cc:485] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:34.687261 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:34.711215 371 consensus_peers.cc:485] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:34.730782 370 consensus_peers.cc:485] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:34.776513 32708 consensus_peers.cc:485] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:34.801916 371 consensus_peers.cc:485] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:34.814107 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:34.815337 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:34.838389 17305 raft_consensus.cc:482] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:34.838567 17305 raft_consensus.cc:504] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:15:34.839146 17305 leader_election.cc:288] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
W20240503 02:15:34.840540 30070 leader_election.cc:334] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:34.840955 30069 leader_election.cc:334] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:34.841074 30069 leader_election.cc:302] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:34.841259 17305 raft_consensus.cc:2726] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:34.864708 29939 consensus_peers.cc:485] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:34.881827 29809 consensus_peers.cc:485] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:34.943634 30070 consensus_peers.cc:485] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:34.945271 30200 consensus_peers.cc:485] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:34.946856 32708 consensus_peers.cc:485] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:34.951339 29940 consensus_peers.cc:485] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:34.960186 32708 consensus_peers.cc:485] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:35.022486 32708 consensus_peers.cc:485] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:35.025668 30200 consensus_peers.cc:485] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:35.042413 29940 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:35.065889 32709 consensus_peers.cc:485] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:35.087986 32709 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:35.118912 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:35.126183 30070 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:35.164786 30069 consensus_peers.cc:485] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:35.213713 370 consensus_peers.cc:485] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:35.232172 30199 consensus_peers.cc:485] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:35.247126 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:35.249572 30070 consensus_peers.cc:485] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:35.297963 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:35.310678 29808 consensus_peers.cc:485] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:35.370282 30069 consensus_peers.cc:485] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:35.417131 30199 consensus_peers.cc:485] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:35.422927 29809 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:35.487310 29808 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:35.548611 370 consensus_peers.cc:485] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:35.552736 29808 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:35.559254 30070 consensus_peers.cc:485] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:35.572402 30199 consensus_peers.cc:485] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:35.584409 30069 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:35.610401 29940 consensus_peers.cc:485] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:35.632400 29809 consensus_peers.cc:485] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:35.695050 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:35.741626 32708 consensus_peers.cc:485] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:35.776760 32708 consensus_peers.cc:485] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:35.783670 32708 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:35.818377 30199 consensus_peers.cc:485] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:35.826421 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:35.829675 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:35.864480 30200 consensus_peers.cc:485] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:35.871297 370 consensus_peers.cc:485] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:35.959903 30199 consensus_peers.cc:485] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:36.000562 371 consensus_peers.cc:485] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:36.012048 29809 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:36.027570 370 consensus_peers.cc:485] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:36.092017 29809 consensus_peers.cc:485] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:36.129833 30199 consensus_peers.cc:485] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:36.132119 29808 consensus_peers.cc:485] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:36.169847 29939 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:36.227020 30069 consensus_peers.cc:485] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:15:36.262899 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:36.289513 370 consensus_peers.cc:485] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:36.295908 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:36.352104 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:36.386750 30199 consensus_peers.cc:485] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:36.438840 29808 consensus_peers.cc:485] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:36.450273 30069 consensus_peers.cc:485] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:36.455708 30070 consensus_peers.cc:485] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:36.495719 370 consensus_peers.cc:485] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:36.505645 29809 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:36.508698 30070 consensus_peers.cc:485] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:36.608269 29809 consensus_peers.cc:485] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:36.657485 29809 consensus_peers.cc:485] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:36.663868 371 consensus_peers.cc:485] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:36.668435 30070 consensus_peers.cc:485] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:36.729775 30200 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:36.736245 30199 consensus_peers.cc:485] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:36.742985 29808 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:36.762370 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:36.768175 371 consensus_peers.cc:485] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:36.776248 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:36.837811 30200 consensus_peers.cc:485] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:36.899379 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:36.919804 30200 consensus_peers.cc:485] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:36.920226 30199 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:36.980616 30069 consensus_peers.cc:485] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:37.058773 29808 consensus_peers.cc:485] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:37.081250 30070 consensus_peers.cc:485] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:37.088320 30070 consensus_peers.cc:485] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:37.108880 29940 consensus_peers.cc:485] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:37.165886 30200 consensus_peers.cc:485] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:37.185448 370 consensus_peers.cc:485] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:37.210459 371 consensus_peers.cc:485] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:37.222041 32708 consensus_peers.cc:485] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:37.280810 29809 consensus_peers.cc:485] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:37.300343 371 consensus_peers.cc:485] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:37.311431 30200 consensus_peers.cc:485] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:37.314049 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:37.325325 32709 consensus_peers.cc:485] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:37.337370 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:37.387491 30069 consensus_peers.cc:485] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:37.388726 32708 consensus_peers.cc:485] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:37.402709 30070 consensus_peers.cc:485] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:37.443560 29940 consensus_peers.cc:485] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:37.450484 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:37.524448 32708 consensus_peers.cc:485] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:37.545158 29939 consensus_peers.cc:485] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:37.608263 30070 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:37.640429 30069 consensus_peers.cc:485] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:37.642367 32709 consensus_peers.cc:485] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:37.673231 29940 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:37.702134 32708 consensus_peers.cc:485] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:37.732969 29808 consensus_peers.cc:485] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:37.735589 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:37.764657 29809 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 149 similar messages]
W20240503 02:15:37.804652 30069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111) [suppressed 151 similar messages]
W20240503 02:15:37.831982 32709 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:37.912395 30070 consensus_peers.cc:485] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:37.915625 370 consensus_peers.cc:485] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:37.943078 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:37.944612 30200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 156 similar messages]
W20240503 02:15:37.974205 30199 consensus_peers.cc:485] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:37.978078 30199 consensus_peers.cc:485] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:37.996330 29809 consensus_peers.cc:485] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:38.025729 370 consensus_peers.cc:485] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:38.036471 29809 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:38.057354 29808 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:38.062690 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:38.066666 30199 consensus_peers.cc:485] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:38.143651 29940 consensus_peers.cc:485] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:38.154259 29808 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:38.187587 29939 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240503 02:15:38.214862 30069 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:38.246778 371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240503 02:15:38.262028 32708 consensus_peers.cc:485] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:38.282384 32708 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:38.299584 30070 consensus_peers.cc:485] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:38.329335 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:38.338476 29809 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:38.345135 30199 consensus_peers.cc:485] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:38.345909 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:38.347031 30200 consensus_peers.cc:485] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:38.390190 32708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111) [suppressed 104 similar messages]
W20240503 02:15:38.391252 32708 consensus_peers.cc:485] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:38.414216 370 consensus_peers.cc:485] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:38.420423 371 consensus_peers.cc:485] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:15:38.459148 370 consensus_peers.cc:485] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:38.531975 30199 consensus_peers.cc:485] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:38.579123 29808 consensus_peers.cc:485] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:38.631711 29939 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:38.640799 30199 consensus_peers.cc:485] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:38.656237 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:38.671684 30069 consensus_peers.cc:485] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:15:38.694629 29809 consensus_peers.cc:485] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:38.756080 30069 consensus_peers.cc:485] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:38.841181 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:38.855927 370 consensus_peers.cc:485] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:38.874745 29808 consensus_peers.cc:485] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:38.897758 29809 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:38.912709 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:39.117537 370 consensus_peers.cc:485] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:39.129531 30070 consensus_peers.cc:485] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:39.141750 30199 consensus_peers.cc:485] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:39.157433 371 consensus_peers.cc:485] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:15:39.159762 30070 consensus_peers.cc:485] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:39.206779 29808 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:39.241600 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:39.253407 29809 consensus_peers.cc:485] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:15:39.297705 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:39.317265 30070 consensus_peers.cc:485] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:39.325582 371 consensus_peers.cc:485] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:39.343201 29809 consensus_peers.cc:485] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:39.353783 30199 consensus_peers.cc:485] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:39.385735 30070 consensus_peers.cc:485] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:15:39.388469 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:39.461841 29940 consensus_peers.cc:485] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:15:39.469568 32708 consensus_peers.cc:485] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:39.488327 30200 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:39.529441 30200 consensus_peers.cc:485] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:39.574276 29809 consensus_peers.cc:485] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:15:39.629068 30199 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:39.654105 30200 consensus_peers.cc:485] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:39.655422 370 consensus_peers.cc:485] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:39.666132 30200 consensus_peers.cc:485] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:15:39.667331 30070 consensus_peers.cc:485] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:15:39.681829 30069 consensus_peers.cc:485] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:39.723008 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:39.726478 29808 consensus_peers.cc:485] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:39.730075 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:39.748644 32708 consensus_peers.cc:485] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:39.774346 32709 consensus_peers.cc:485] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:39.790556 371 consensus_peers.cc:485] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:15:39.808148 30070 consensus_peers.cc:485] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:15:39.859637 371 consensus_peers.cc:485] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:39.871978 30200 consensus_peers.cc:485] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:15:39.889149 29940 consensus_peers.cc:485] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:39.890071 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:39.935115 29939 consensus_peers.cc:485] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:39.946377 30069 consensus_peers.cc:485] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:40.025120 32709 consensus_peers.cc:485] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:15:40.089434 32708 consensus_peers.cc:485] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:40.090590 30070 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:15:40.154009 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:40.178287 32708 consensus_peers.cc:485] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:40.185269 29940 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:15:40.194212 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:40.268087 30069 consensus_peers.cc:485] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:40.363359 29808 consensus_peers.cc:485] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:40.423147 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:40.438766 370 consensus_peers.cc:485] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:40.447420 29808 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 02:15:40.458636 17334 raft_consensus.cc:482] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:40.458858 17334 raft_consensus.cc:504] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:15:40.459580 17334 leader_election.cc:288] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:15:40.460842 30069 leader_election.cc:334] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:40.461153 30070 leader_election.cc:334] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:40.461292 30070 leader_election.cc:302] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:40.461804 17334 raft_consensus.cc:2726] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:40.463012 29809 consensus_peers.cc:485] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:15:40.481256 30070 consensus_peers.cc:485] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:40.515664 30199 consensus_peers.cc:485] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:40.544131 32709 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:15:40.565737 30199 consensus_peers.cc:485] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:40.589445 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:40.612568 370 consensus_peers.cc:485] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:40.637543 30070 consensus_peers.cc:485] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:15:40.666133 29809 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:15:40.701617 32708 consensus_peers.cc:485] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:40.723604 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:40.731775 29808 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:40.783730 29940 consensus_peers.cc:485] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:15:40.787217 30199 consensus_peers.cc:485] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:40.788972 30069 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:40.791010 32708 consensus_peers.cc:485] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:40.799621 370 consensus_peers.cc:485] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:40.867183 30199 consensus_peers.cc:485] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:40.896277 30200 consensus_peers.cc:485] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:15:40.910315 29939 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 02:15:40.977957 17313 raft_consensus.cc:482] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:40.978133 17313 raft_consensus.cc:504] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:15:40.979250 17313 leader_election.cc:288] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:15:40.980437 371 leader_election.cc:334] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:40.980809 370 leader_election.cc:334] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:40.980934 370 leader_election.cc:302] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:40.981209 17313 raft_consensus.cc:2726] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:40.991930 371 consensus_peers.cc:485] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:15:41.020102 32708 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:41.023854 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:41.044270 370 consensus_peers.cc:485] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:41.050021 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:41.071524 29809 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:15:41.096604 30199 consensus_peers.cc:485] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:41.154525 29808 consensus_peers.cc:485] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:41.178040 370 consensus_peers.cc:485] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:41.226675 30199 consensus_peers.cc:485] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:41.240046 30069 consensus_peers.cc:485] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:15:41.246687 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:41.313002 29809 consensus_peers.cc:485] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:15:41.437837 29808 consensus_peers.cc:485] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:41.457242 30069 consensus_peers.cc:485] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:41.478358 29809 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:15:41.519847 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:41.556704 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:41.569986 30199 consensus_peers.cc:485] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:41.644291 30070 consensus_peers.cc:485] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:15:41.658299 30070 consensus_peers.cc:485] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:41.659802 29809 consensus_peers.cc:485] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:15:41.721139 29808 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:41.724040 29940 consensus_peers.cc:485] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:15:41.753566 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:41.768347 29809 consensus_peers.cc:485] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:15:41.774128 370 consensus_peers.cc:485] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:41.774816 30199 consensus_peers.cc:485] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:41.780398 30070 consensus_peers.cc:485] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:15:41.848062 30070 consensus_peers.cc:485] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:15:41.929095 371 consensus_peers.cc:485] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:15:41.971195 30200 consensus_peers.cc:485] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:15:42.041961 371 consensus_peers.cc:485] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:15:42.067463 371 consensus_peers.cc:485] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:15:42.069967 32708 consensus_peers.cc:485] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:42.075361 30070 consensus_peers.cc:485] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:15:42.087896 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:42.106863 29809 consensus_peers.cc:485] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:15:42.108254 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:42.177505 30200 consensus_peers.cc:485] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:15:42.202028 30070 consensus_peers.cc:485] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:15:42.239077 32708 consensus_peers.cc:485] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:42.244580 30200 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:15:42.248394 30199 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:42.256841 29940 consensus_peers.cc:485] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:15:42.284328 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:42.290123 32709 consensus_peers.cc:485] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:15:42.401583 370 consensus_peers.cc:485] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:42.404876 30200 consensus_peers.cc:485] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:42.427299 371 consensus_peers.cc:485] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:15:42.430428 29939 consensus_peers.cc:485] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:42.473557 30069 consensus_peers.cc:485] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:42.519587 29808 consensus_peers.cc:485] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:42.539199 32708 consensus_peers.cc:485] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:42.564991 30070 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:15:42.596565 30200 consensus_peers.cc:485] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:15:42.629069 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:42.639658 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:42.655952 30069 consensus_peers.cc:485] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:42.689707 32709 consensus_peers.cc:485] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:15:42.697351 32708 consensus_peers.cc:485] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:42.742381 29940 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:15:42.778787 29808 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111) [suppressed 148 similar messages]
W20240503 02:15:42.804733 30069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111) [suppressed 148 similar messages]
W20240503 02:15:42.890592 30069 consensus_peers.cc:485] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:42.890977 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:42.913605 370 consensus_peers.cc:485] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:42.920485 29808 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:42.945201 30200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 158 similar messages]
W20240503 02:15:42.955070 30199 consensus_peers.cc:485] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:42.957612 370 consensus_peers.cc:485] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:42.959414 32708 consensus_peers.cc:485] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:42.991091 29809 consensus_peers.cc:485] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:15:43.013243 370 consensus_peers.cc:485] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:43.034716 30070 consensus_peers.cc:485] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:43.050843 30199 consensus_peers.cc:485] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:43.084223 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:43.087587 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:43.114711 30199 consensus_peers.cc:485] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:43.116900 30200 consensus_peers.cc:485] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:15:43.167448 32709 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:15:43.174975 29808 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:43.196754 29940 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240503 02:15:43.204383 29808 consensus_peers.cc:485] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:43.223424 29940 consensus_peers.cc:485] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:15:43.226647 32708 consensus_peers.cc:485] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:43.235255 30199 consensus_peers.cc:485] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:43.278517 29809 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:15:43.298595 30069 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 02:15:43.326165 353 maintenance_manager.cc:419] P 8a666e798fe8477d8dd1e51b385d5ead: Scheduling FlushMRSOp(db0756f0f331402aaef0583e1f9e5318): perf score=0.033349
W20240503 02:15:43.335769 30070 consensus_peers.cc:485] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:15:43.357247 370 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240503 02:15:43.386693 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:43.400055 32712 maintenance_manager.cc:643] P 8a666e798fe8477d8dd1e51b385d5ead: FlushMRSOp(db0756f0f331402aaef0583e1f9e5318) complete. Timing: real 0.074s user 0.051s sys 0.017s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":297,"dirs.run_cpu_time_us":167,"dirs.run_wall_time_us":981,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":596,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":17771,"threads_started":2,"wal-append.queue_time_us":17653}
I20240503 02:15:43.402025 353 maintenance_manager.cc:419] P 8a666e798fe8477d8dd1e51b385d5ead: Scheduling UndoDeltaBlockGCOp(db0756f0f331402aaef0583e1f9e5318): 4579 bytes on disk
I20240503 02:15:43.402644 32712 maintenance_manager.cc:643] P 8a666e798fe8477d8dd1e51b385d5ead: UndoDeltaBlockGCOp(db0756f0f331402aaef0583e1f9e5318) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
W20240503 02:15:43.406250 29939 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:43.431269 32709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240503 02:15:43.474939 32708 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:43.535820 30199 consensus_peers.cc:485] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:43.586830 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:43.628525 370 consensus_peers.cc:485] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:43.630800 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:43.681067 29808 consensus_peers.cc:485] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:43.681056 370 consensus_peers.cc:485] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:43.702054 29809 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:15:43.749816 371 consensus_peers.cc:485] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:15:43.764437 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:43.847764 29809 consensus_peers.cc:485] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:15:43.848716 29808 consensus_peers.cc:485] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:43.853049 30199 consensus_peers.cc:485] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:43.882161 30069 consensus_peers.cc:485] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:15:43.963508 30199 consensus_peers.cc:485] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:43.985061 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:44.015731 370 consensus_peers.cc:485] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 02:15:44.061126 17338 raft_consensus.cc:482] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:44.061332 17338 raft_consensus.cc:504] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:15:44.062086 17338 leader_election.cc:288] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:15:44.063251 371 leader_election.cc:334] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:44.063474 370 leader_election.cc:334] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:44.063621 370 leader_election.cc:302] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:44.063854 17338 raft_consensus.cc:2726] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:44.070401 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:44.073624 30069 consensus_peers.cc:485] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:44.083932 29809 consensus_peers.cc:485] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:15:44.089134 29940 consensus_peers.cc:485] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:15:44.101496 30070 consensus_peers.cc:485] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:44.144119 29809 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:15:44.181955 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:44.195103 29808 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:44.223999 29809 consensus_peers.cc:485] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:15:44.274912 30070 consensus_peers.cc:485] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:15:44.356097 30200 consensus_peers.cc:485] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:15:44.418262 371 consensus_peers.cc:485] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:15:44.423389 30070 consensus_peers.cc:485] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:15:44.427708 30200 consensus_peers.cc:485] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:15:44.456590 371 consensus_peers.cc:485] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:15:44.474450 30070 consensus_peers.cc:485] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:15:44.495159 30199 consensus_peers.cc:485] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:44.496855 371 consensus_peers.cc:485] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 02:15:44.504503 17355 raft_consensus.cc:482] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:44.504663 17355 raft_consensus.cc:504] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:15:44.505407 17355 leader_election.cc:288] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:15:44.506688 29808 leader_election.cc:334] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:44.506882 29809 leader_election.cc:334] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:44.506990 29809 leader_election.cc:302] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 16440e365430405fb10366ab70e2b429; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:44.507228 17355 raft_consensus.cc:2726] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:44.522420 29809 consensus_peers.cc:485] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:15:44.569272 32708 consensus_peers.cc:485] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:44.598265 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:44.621589 29940 consensus_peers.cc:485] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:15:44.632021 32708 consensus_peers.cc:485] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:44.642468 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:44.652907 30070 consensus_peers.cc:485] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:15:44.744709 30200 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:15:44.745149 32709 consensus_peers.cc:485] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:15:44.777269 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:44.809649 370 consensus_peers.cc:485] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:44.847191 30199 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:44.855051 32709 consensus_peers.cc:485] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:15:44.881079 371 consensus_peers.cc:485] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:15:44.912602 30070 consensus_peers.cc:485] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:15:44.934612 30200 consensus_peers.cc:485] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:15:44.999653 29939 consensus_peers.cc:485] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:45.009598 30200 consensus_peers.cc:485] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:45.018076 30070 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:15:45.030964 30069 consensus_peers.cc:485] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:45.069494 29940 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:15:45.112138 32708 consensus_peers.cc:485] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:45.160485 30069 consensus_peers.cc:485] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:45.178694 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:45.184958 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:45.201074 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:45.313181 29808 consensus_peers.cc:485] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:45.375418 32708 consensus_peers.cc:485] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:45.407112 370 consensus_peers.cc:485] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:45.485744 29808 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:45.504664 370 consensus_peers.cc:485] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:45.516996 32708 consensus_peers.cc:485] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:45.548112 30199 consensus_peers.cc:485] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:45.550032 30199 consensus_peers.cc:485] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:45.571988 32708 consensus_peers.cc:485] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:45.575737 30199 consensus_peers.cc:485] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:45.577785 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:45.584614 29808 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:45.592476 30199 consensus_peers.cc:485] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:45.620899 30070 consensus_peers.cc:485] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:45.639377 32709 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:15:45.673998 370 consensus_peers.cc:485] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:45.704908 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:45.711480 30199 consensus_peers.cc:485] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:45.713392 30069 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:45.720201 29808 consensus_peers.cc:485] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:45.747092 29809 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:15:45.747982 30070 consensus_peers.cc:485] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:15:45.764845 29809 consensus_peers.cc:485] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:15:45.773800 30069 consensus_peers.cc:485] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:45.775703 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:45.838573 29940 consensus_peers.cc:485] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:15:45.925715 32708 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:45.965560 30200 consensus_peers.cc:485] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:15:45.968526 370 consensus_peers.cc:485] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:45.996330 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:46.090970 371 consensus_peers.cc:485] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:15:46.093145 29939 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:46.172981 29808 consensus_peers.cc:485] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:46.189214 370 consensus_peers.cc:485] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:46.219872 29809 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:15:46.232829 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:46.247304 29808 consensus_peers.cc:485] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:46.251641 30199 consensus_peers.cc:485] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 02:15:46.259127 17338 raft_consensus.cc:482] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:46.259274 17338 raft_consensus.cc:504] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:15:46.259797 17338 leader_election.cc:288] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:15:46.261076 371 leader_election.cc:334] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:46.261361 370 leader_election.cc:334] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:46.261479 370 leader_election.cc:302] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:46.261761 17338 raft_consensus.cc:2726] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:46.290580 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:46.388993 370 consensus_peers.cc:485] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:46.400528 29809 consensus_peers.cc:485] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:15:46.457422 30199 consensus_peers.cc:485] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:46.520279 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:46.576141 29809 consensus_peers.cc:485] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:15:46.581918 29808 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:46.596697 29809 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:15:46.632193 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:46.636545 30069 consensus_peers.cc:485] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:15:46.646153 30070 consensus_peers.cc:485] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:46.660362 30070 consensus_peers.cc:485] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:15:46.671325 29940 consensus_peers.cc:485] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:15:46.678964 29809 consensus_peers.cc:485] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:15:46.745255 30069 consensus_peers.cc:485] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:46.774554 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:46.843309 30070 consensus_peers.cc:485] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:15:46.849678 30200 consensus_peers.cc:485] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:15:46.914588 29809 consensus_peers.cc:485] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:15:46.986303 30070 consensus_peers.cc:485] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:15:46.999361 371 consensus_peers.cc:485] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:15:47.041139 371 consensus_peers.cc:485] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:15:47.045941 371 consensus_peers.cc:485] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:15:47.108290 29940 consensus_peers.cc:485] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:15:47.113008 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:47.122888 32708 consensus_peers.cc:485] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:47.139425 30070 consensus_peers.cc:485] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:15:47.153048 30200 consensus_peers.cc:485] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:15:47.153048 30199 consensus_peers.cc:485] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:47.163468 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:47.167799 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:47.219933 30200 consensus_peers.cc:485] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:15:47.269781 32708 consensus_peers.cc:485] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:47.301167 370 consensus_peers.cc:485] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 02:15:47.385977 17374 raft_consensus.cc:482] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:47.386142 17374 raft_consensus.cc:504] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:15:47.386734 17374 leader_election.cc:288] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:15:47.388268 30069 leader_election.cc:334] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:47.389564 30070 leader_election.cc:334] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:47.389734 30070 leader_election.cc:302] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:47.390012 17374 raft_consensus.cc:2726] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:47.447278 30069 consensus_peers.cc:485] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:47.459100 30200 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:15:47.460742 371 consensus_peers.cc:485] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:15:47.488631 32709 consensus_peers.cc:485] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:15:47.489751 30070 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:15:47.532357 29939 consensus_peers.cc:485] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:47.536091 32709 consensus_peers.cc:485] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:15:47.573354 29940 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:15:47.605420 30199 consensus_peers.cc:485] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:47.621093 30199 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:47.622717 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:47.653561 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:47.657409 30070 consensus_peers.cc:485] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 02:15:47.716231 17358 raft_consensus.cc:482] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:47.716449 17358 raft_consensus.cc:504] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:15:47.717242 17358 leader_election.cc:288] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
W20240503 02:15:47.718561 32709 leader_election.cc:334] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:47.718991 32708 leader_election.cc:334] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:47.719105 32708 leader_election.cc:302] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:47.719516 17358 raft_consensus.cc:2726] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:15:47.746613 30200 consensus_peers.cc:485] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:47.748224 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:47.788502 29809 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 151 similar messages]
W20240503 02:15:47.795699 32708 consensus_peers.cc:485] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:47.797570 30069 consensus_peers.cc:485] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:47.813148 30069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111) [suppressed 149 similar messages]
W20240503 02:15:47.813877 370 consensus_peers.cc:485] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:47.814934 29808 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:47.872943 30070 consensus_peers.cc:485] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:47.883936 30199 consensus_peers.cc:485] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:47.958165 30200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 160 similar messages]
W20240503 02:15:47.962194 29809 consensus_peers.cc:485] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:15:47.992489 370 consensus_peers.cc:485] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:47.992839 30199 consensus_peers.cc:485] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:48.000360 32708 consensus_peers.cc:485] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:48.006318 32708 consensus_peers.cc:485] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:48.019855 29808 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:48.031698 29808 consensus_peers.cc:485] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:48.054139 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:48.059445 29809 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:15:48.071950 30070 consensus_peers.cc:485] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:15:48.076184 370 consensus_peers.cc:485] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:48.108999 32708 consensus_peers.cc:485] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:48.185242 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:48.202783 29940 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240503 02:15:48.203642 29940 consensus_peers.cc:485] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:15:48.225418 30199 consensus_peers.cc:485] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:48.230690 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:48.247009 32709 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:15:48.248853 370 consensus_peers.cc:485] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:48.284240 30199 consensus_peers.cc:485] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:48.301482 30069 consensus_peers.cc:485] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:48.303138 32708 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:48.354228 29808 consensus_peers.cc:485] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:48.355702 30200 consensus_peers.cc:485] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:15:48.385850 370 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111) [suppressed 125 similar messages]
W20240503 02:15:48.399871 30069 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:48.552556 29939 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:48.562569 32709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240503 02:15:48.599099 30199 consensus_peers.cc:485] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:48.623469 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:48.639065 29809 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:15:48.644057 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:48.676434 29808 consensus_peers.cc:485] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:48.694085 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:48.781123 371 consensus_peers.cc:485] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:15:48.792054 370 consensus_peers.cc:485] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:48.802191 29808 consensus_peers.cc:485] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:48.805876 370 consensus_peers.cc:485] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:48.826133 30199 consensus_peers.cc:485] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:48.848109 29809 consensus_peers.cc:485] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:15:48.875851 30070 consensus_peers.cc:485] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:48.902812 29809 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:15:48.978422 30069 consensus_peers.cc:485] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:15:49.042804 29809 consensus_peers.cc:485] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:15:49.071398 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:49.079015 29808 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:49.129422 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:49.172869 30200 consensus_peers.cc:485] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:15:49.253515 29809 consensus_peers.cc:485] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:15:49.273013 29809 consensus_peers.cc:485] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:15:49.288823 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:49.335094 30070 consensus_peers.cc:485] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:15:49.358070 30069 consensus_peers.cc:485] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:49.364740 371 consensus_peers.cc:485] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:15:49.373981 29940 consensus_peers.cc:485] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:15:49.443318 371 consensus_peers.cc:485] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 02:15:49.445866 353 maintenance_manager.cc:419] P 8a666e798fe8477d8dd1e51b385d5ead: Scheduling FlushMRSOp(b7a2201f9605454ea12df4e3a7e5a63e): perf score=0.033351
W20240503 02:15:49.455013 30070 consensus_peers.cc:485] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:15:49.463135 30070 consensus_peers.cc:485] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 02:15:49.496461 32712 maintenance_manager.cc:643] P 8a666e798fe8477d8dd1e51b385d5ead: FlushMRSOp(b7a2201f9605454ea12df4e3a7e5a63e) complete. Timing: real 0.050s user 0.034s sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":312,"dirs.run_cpu_time_us":212,"dirs.run_wall_time_us":1459,"drs_written":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4,"lbm_write_time_us":677,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":15163,"threads_started":2,"wal-append.queue_time_us":15043}
I20240503 02:15:49.498373 353 maintenance_manager.cc:419] P 8a666e798fe8477d8dd1e51b385d5ead: Scheduling UndoDeltaBlockGCOp(b7a2201f9605454ea12df4e3a7e5a63e): 4838 bytes on disk
I20240503 02:15:49.498871 32712 maintenance_manager.cc:643] P 8a666e798fe8477d8dd1e51b385d5ead: UndoDeltaBlockGCOp(b7a2201f9605454ea12df4e3a7e5a63e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
W20240503 02:15:49.521903 30200 consensus_peers.cc:485] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:15:49.682536 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:49.713728 30070 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:15:49.717666 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:49.747133 30200 consensus_peers.cc:485] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:15:49.765007 371 consensus_peers.cc:485] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:15:49.771301 29940 consensus_peers.cc:485] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:15:49.782902 30199 consensus_peers.cc:485] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:49.783711 371 consensus_peers.cc:485] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:15:49.796434 32708 consensus_peers.cc:485] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:49.799548 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:49.831949 30070 consensus_peers.cc:485] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:15:49.853843 32708 consensus_peers.cc:485] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:15:49.877743 17374 raft_consensus.cc:482] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:49.877969 17374 raft_consensus.cc:504] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:15:49.878732 17374 leader_election.cc:288] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
W20240503 02:15:49.880977 30069 leader_election.cc:334] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:49.882284 30070 leader_election.cc:334] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:49.882408 30070 leader_election.cc:302] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:49.882751 17374 raft_consensus.cc:2726] T 9be5395489a3488da62f4d5b66facf3d P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:49.969089 29939 consensus_peers.cc:485] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:49.982340 30070 consensus_peers.cc:485] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:15:50.032709 32708 consensus_peers.cc:485] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:50.035624 30200 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:15:50.044574 32709 consensus_peers.cc:485] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:15:50.048009 32709 consensus_peers.cc:485] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:15:50.069790 30069 consensus_peers.cc:485] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:50.124944 29940 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:15:50.191591 29808 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:50.221956 32708 consensus_peers.cc:485] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:50.228992 30200 consensus_peers.cc:485] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:50.231431 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:50.236716 30199 consensus_peers.cc:485] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:50.238564 29808 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:50.244402 370 consensus_peers.cc:485] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:50.246306 30069 consensus_peers.cc:485] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:50.279294 29809 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:15:50.286692 370 consensus_peers.cc:485] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:50.305392 370 consensus_peers.cc:485] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:50.312814 30199 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:50.317605 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:50.331604 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:50.371933 30199 consensus_peers.cc:485] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:50.461982 29940 consensus_peers.cc:485] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:15:50.464692 29809 consensus_peers.cc:485] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:15:50.475558 30070 consensus_peers.cc:485] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:50.483886 30070 consensus_peers.cc:485] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:15:50.547468 32708 consensus_peers.cc:485] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:50.556349 32708 consensus_peers.cc:485] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:50.579583 370 consensus_peers.cc:485] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:50.629685 30199 consensus_peers.cc:485] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:50.677557 29808 consensus_peers.cc:485] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:50.708786 30199 consensus_peers.cc:485] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:50.710434 30199 consensus_peers.cc:485] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:50.795593 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:50.806126 32708 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:50.860013 370 consensus_peers.cc:485] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:50.868827 32709 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:15:50.905606 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:50.939227 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:51.011077 29939 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:51.012238 30069 consensus_peers.cc:485] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:51.014079 370 consensus_peers.cc:485] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:51.058706 30200 consensus_peers.cc:485] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:15:51.107198 29808 consensus_peers.cc:485] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:51.150007 30069 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:51.164372 30199 consensus_peers.cc:485] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:51.177657 29809 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:15:51.180423 29808 consensus_peers.cc:485] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:51.184631 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:51.195071 30199 consensus_peers.cc:485] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:51.261008 370 consensus_peers.cc:485] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:51.302628 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:51.305781 371 consensus_peers.cc:485] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:15:51.317948 30069 consensus_peers.cc:485] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:15:51.340044 29808 consensus_peers.cc:485] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:51.385543 29809 consensus_peers.cc:485] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:15:51.396662 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:51.414072 30070 consensus_peers.cc:485] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:51.581786 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:51.606705 29809 consensus_peers.cc:485] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:15:51.610286 29808 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:51.617000 371 consensus_peers.cc:485] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:15:51.651108 29809 consensus_peers.cc:485] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:15:51.662324 29809 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:15:51.684495 30200 consensus_peers.cc:485] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 02:15:51.684600 17382 raft_consensus.cc:482] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:51.684777 17382 raft_consensus.cc:504] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:15:51.685474 17382 leader_election.cc:288] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:15:51.686784 371 leader_election.cc:334] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:51.687151 370 leader_election.cc:334] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:51.687273 370 leader_election.cc:302] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:51.687520 17382 raft_consensus.cc:2726] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:51.690582 30069 consensus_peers.cc:485] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:51.690703 29940 consensus_peers.cc:485] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:15:51.734207 30070 consensus_peers.cc:485] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:15:51.736210 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:51.808239 29809 consensus_peers.cc:485] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:15:51.931631 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:52.004694 371 consensus_peers.cc:485] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:15:52.012431 30070 consensus_peers.cc:485] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:15:52.041101 30070 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:15:52.055581 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:52.106715 30200 consensus_peers.cc:485] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:15:52.140664 30070 consensus_peers.cc:485] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:15:52.193512 29940 consensus_peers.cc:485] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:15:52.205842 29939 consensus_peers.cc:485] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:52.220176 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:52.289994 30070 consensus_peers.cc:485] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:15:52.291002 371 consensus_peers.cc:485] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:15:52.335966 32708 consensus_peers.cc:485] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:52.359163 32709 consensus_peers.cc:485] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:15:52.367411 371 consensus_peers.cc:485] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:15:52.380620 30200 consensus_peers.cc:485] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:15:52.397668 30200 consensus_peers.cc:485] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:52.426324 29808 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:52.429054 32709 consensus_peers.cc:485] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:15:52.434185 30199 consensus_peers.cc:485] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:52.465402 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:15:52.491112 17395 raft_consensus.cc:482] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:52.491411 17395 raft_consensus.cc:504] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:15:52.492357 17395 leader_election.cc:288] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:15:52.493736 29809 leader_election.cc:334] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:52.495025 29808 leader_election.cc:334] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:52.495206 29808 leader_election.cc:302] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 16440e365430405fb10366ab70e2b429; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:52.495939 17395 raft_consensus.cc:2726] T 6bd4fcc4faf44d148a03beaafc1c05ba P 16440e365430405fb10366ab70e2b429 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:52.504922 32708 consensus_peers.cc:485] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:52.522181 30200 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:15:52.530301 30070 consensus_peers.cc:485] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:15:52.562949 30069 consensus_peers.cc:485] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:52.565476 30199 consensus_peers.cc:485] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:52.567138 30199 consensus_peers.cc:485] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:52.593443 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:52.610188 29808 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:52.612704 32708 consensus_peers.cc:485] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:52.683079 370 consensus_peers.cc:485] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:52.714066 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:52.751895 370 consensus_peers.cc:485] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:52.768168 29940 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:15:52.811762 30199 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:52.828913 29809 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 152 similar messages]
W20240503 02:15:52.855329 370 consensus_peers.cc:485] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:52.866456 29809 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:15:52.868278 32708 consensus_peers.cc:485] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:52.884758 32708 consensus_peers.cc:485] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:52.887580 30069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111) [suppressed 152 similar messages]
W20240503 02:15:52.930068 30070 consensus_peers.cc:485] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:52.962982 30199 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111) [suppressed 158 similar messages]
W20240503 02:15:52.964226 30199 consensus_peers.cc:485] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:52.974881 30069 consensus_peers.cc:485] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:53.032815 30070 consensus_peers.cc:485] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:15:53.040009 29808 consensus_peers.cc:485] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:53.040513 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:53.057998 370 consensus_peers.cc:485] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:53.062266 32708 consensus_peers.cc:485] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:53.070690 29809 consensus_peers.cc:485] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:15:53.099483 370 consensus_peers.cc:485] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:53.108345 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:53.160193 30199 consensus_peers.cc:485] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:53.161590 30199 consensus_peers.cc:485] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:53.214970 29940 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240503 02:15:53.216714 29940 consensus_peers.cc:485] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:15:53.226989 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:53.333135 29808 consensus_peers.cc:485] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:53.340575 32708 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:53.464545 29808 consensus_peers.cc:485] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:53.557950 29809 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:15:53.559074 29939 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:53.561868 30069 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:53.588181 30069 consensus_peers.cc:485] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 02:15:53.589939 17409 raft_consensus.cc:482] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:53.590116 17409 raft_consensus.cc:504] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:15:53.590726 17409 leader_election.cc:288] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:15:53.592448 30070 leader_election.cc:334] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:53.592985 30069 leader_election.cc:334] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:53.593143 30069 leader_election.cc:302] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a9094240860e43bfb129193f3bfd5be6; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:53.593600 17409 raft_consensus.cc:2726] T 36d3f1edbab7477f9b7ac400160491a0 P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:53.607282 32708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240503 02:15:53.611358 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:53.611588 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:53.630508 370 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111) [suppressed 125 similar messages]
W20240503 02:15:53.632411 370 consensus_peers.cc:485] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:53.633517 30199 consensus_peers.cc:485] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:53.647703 370 consensus_peers.cc:485] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:53.653471 30199 consensus_peers.cc:485] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:53.734375 32709 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:15:53.739192 371 consensus_peers.cc:485] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:15:53.741839 30069 consensus_peers.cc:485] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:15:53.748380 30200 consensus_peers.cc:485] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:15:53.759884 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:53.905974 29809 consensus_peers.cc:485] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:15:53.936389 29808 consensus_peers.cc:485] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:53.939245 29809 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:15:54.028129 29809 consensus_peers.cc:485] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:15:54.055305 29940 consensus_peers.cc:485] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:15:54.080880 30070 consensus_peers.cc:485] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:54.104423 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:54.113364 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:54.170038 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:54.183254 30070 consensus_peers.cc:485] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:15:54.205461 29809 consensus_peers.cc:485] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:15:54.212785 30069 consensus_peers.cc:485] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:54.276608 29808 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:54.299490 371 consensus_peers.cc:485] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:15:54.443122 30070 consensus_peers.cc:485] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:15:54.480134 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:54.495996 29809 consensus_peers.cc:485] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:15:54.519699 30070 consensus_peers.cc:485] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:15:54.523792 30200 consensus_peers.cc:485] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:15:54.537803 30200 consensus_peers.cc:485] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:15:54.559262 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:54.613287 29939 consensus_peers.cc:485] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:54.645608 371 consensus_peers.cc:485] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:15:54.665838 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:54.673944 30070 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:15:54.745110 30070 consensus_peers.cc:485] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:15:54.791067 29940 consensus_peers.cc:485] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:15:54.798198 371 consensus_peers.cc:485] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:15:54.813175 30199 consensus_peers.cc:485] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:54.845706 32708 consensus_peers.cc:485] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:54.873749 32709 consensus_peers.cc:485] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:15:54.911940 30070 consensus_peers.cc:485] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:15:54.926234 30200 consensus_peers.cc:485] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:15:54.927624 29808 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:54.948632 30200 consensus_peers.cc:485] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:54.952597 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:54.959583 32709 consensus_peers.cc:485] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:15:54.976028 371 consensus_peers.cc:485] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:15:54.994077 32708 consensus_peers.cc:485] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:55.069553 29808 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:55.088657 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:55.091084 30199 consensus_peers.cc:485] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:55.096007 30069 consensus_peers.cc:485] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:55.153518 30199 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:55.173893 30199 consensus_peers.cc:485] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:55.185128 29808 consensus_peers.cc:485] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:55.211894 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:55.225214 29940 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:15:55.235214 30070 consensus_peers.cc:485] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:15:55.297163 30199 consensus_peers.cc:485] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:55.307858 32708 consensus_peers.cc:485] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:55.337744 30200 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:15:55.355670 370 consensus_peers.cc:485] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:55.386713 32708 consensus_peers.cc:485] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:55.388559 370 consensus_peers.cc:485] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:55.468955 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:55.472884 30199 consensus_peers.cc:485] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:55.484436 32708 consensus_peers.cc:485] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:55.507100 370 consensus_peers.cc:485] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:55.508123 30070 consensus_peers.cc:485] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:55.526185 30069 consensus_peers.cc:485] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:55.531374 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:55.541703 32708 consensus_peers.cc:485] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:55.596124 32708 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:55.603317 29809 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:15:55.664654 30199 consensus_peers.cc:485] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:55.670665 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:55.740547 29940 consensus_peers.cc:485] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:15:55.743917 29808 consensus_peers.cc:485] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:55.763132 29808 consensus_peers.cc:485] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:55.866353 370 consensus_peers.cc:485] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:55.868956 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:55.899590 29809 consensus_peers.cc:485] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:15:55.934367 370 consensus_peers.cc:485] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:55.935739 370 consensus_peers.cc:485] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:55.998098 29809 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:15:56.041646 30199 consensus_peers.cc:485] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:56.052781 30069 consensus_peers.cc:485] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:56.079546 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:56.125110 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:56.134454 29939 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:56.187749 370 consensus_peers.cc:485] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:56.226292 29809 consensus_peers.cc:485] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:15:56.231597 29808 consensus_peers.cc:485] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:56.259797 32709 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:15:56.305886 30069 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:56.316761 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:56.336937 30069 consensus_peers.cc:485] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:15:56.337235 30200 consensus_peers.cc:485] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:15:56.366865 30070 consensus_peers.cc:485] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:56.384022 371 consensus_peers.cc:485] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:15:56.399358 29809 consensus_peers.cc:485] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:15:56.409837 30199 consensus_peers.cc:485] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:56.417404 29809 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:15:56.439397 29940 consensus_peers.cc:485] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:15:56.459235 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:56.565755 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:56.594985 29808 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:56.760008 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:56.761198 29809 consensus_peers.cc:485] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:15:56.828356 29809 consensus_peers.cc:485] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:15:56.877079 30069 consensus_peers.cc:485] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:56.901587 30200 consensus_peers.cc:485] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:15:56.958081 371 consensus_peers.cc:485] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:15:56.960959 30070 consensus_peers.cc:485] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:15:56.961750 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:56.982172 30070 consensus_peers.cc:485] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:15:57.006872 371 consensus_peers.cc:485] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:15:57.024499 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:57.057571 29939 consensus_peers.cc:485] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:57.093447 371 consensus_peers.cc:485] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:15:57.139281 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:57.142024 30070 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:15:57.150203 30200 consensus_peers.cc:485] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:15:57.197942 32709 consensus_peers.cc:485] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:15:57.223091 29940 consensus_peers.cc:485] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 02:15:57.282284 17429 raft_consensus.cc:482] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:15:57.282450 17429 raft_consensus.cc:504] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:15:57.283038 17429 leader_election.cc:288] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:15:57.284502 371 leader_election.cc:334] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:57.284737 370 leader_election.cc:334] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:15:57.284893 370 leader_election.cc:302] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:15:57.285535 17429 raft_consensus.cc:2726] T ae4373a66af14c8f8a633963dffd2261 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:15:57.313575 30199 consensus_peers.cc:485] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:57.338464 30070 consensus_peers.cc:485] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:15:57.346585 30070 consensus_peers.cc:485] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:15:57.349587 30200 consensus_peers.cc:485] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:15:57.382643 32708 consensus_peers.cc:485] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:57.401742 371 consensus_peers.cc:485] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:15:57.422915 32709 consensus_peers.cc:485] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:15:57.432196 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:57.461086 30199 consensus_peers.cc:485] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:57.535355 30070 consensus_peers.cc:485] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:15:57.541570 32708 consensus_peers.cc:485] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:57.548041 29808 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:57.569326 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:57.573416 29808 consensus_peers.cc:485] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:57.574663 30200 consensus_peers.cc:485] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:57.613409 29808 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:57.623598 30199 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:57.638818 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:57.657853 32708 consensus_peers.cc:485] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:57.687346 30199 consensus_peers.cc:485] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:57.709012 32708 consensus_peers.cc:485] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:57.728045 30199 consensus_peers.cc:485] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:57.813122 32708 consensus_peers.cc:485] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:57.831995 29940 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:15:57.835161 29808 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111) [suppressed 152 similar messages]
W20240503 02:15:57.875141 370 consensus_peers.cc:485] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:57.877599 30069 consensus_peers.cc:485] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:57.904366 30069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111) [suppressed 148 similar messages]
W20240503 02:15:57.905200 30069 consensus_peers.cc:485] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:57.919333 370 consensus_peers.cc:485] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:57.949724 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:57.966572 30200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 162 similar messages]
W20240503 02:15:57.967370 30200 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:15:57.974560 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:57.985141 30070 consensus_peers.cc:485] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:15:58.016700 30199 consensus_peers.cc:485] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:58.039397 370 consensus_peers.cc:485] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:58.109087 32708 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:58.133904 370 consensus_peers.cc:485] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:58.137297 30070 consensus_peers.cc:485] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:58.182777 29809 consensus_peers.cc:485] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:15:58.238631 29940 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240503 02:15:58.239502 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:58.266608 29808 consensus_peers.cc:485] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:58.270831 29809 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:15:58.296376 30199 consensus_peers.cc:485] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:58.303161 32708 consensus_peers.cc:485] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:58.396132 370 consensus_peers.cc:485] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:58.423766 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:58.444473 29808 consensus_peers.cc:485] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:58.459815 29809 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:15:58.488057 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:58.540648 29940 consensus_peers.cc:485] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:15:58.637449 32708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240503 02:15:58.677639 29809 consensus_peers.cc:485] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:15:58.690287 30069 consensus_peers.cc:485] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:58.754978 370 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111) [suppressed 123 similar messages]
W20240503 02:15:58.756033 370 consensus_peers.cc:485] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:58.758952 370 consensus_peers.cc:485] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:58.797112 29808 consensus_peers.cc:485] T 4055b2f8d72041acaa81e6e98db9042c P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:58.801579 30199 consensus_peers.cc:485] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:58.807008 29809 consensus_peers.cc:485] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:15:58.807163 29940 consensus_peers.cc:485] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:15:58.809319 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:58.810899 30069 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:58.849417 29939 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:58.915632 30199 consensus_peers.cc:485] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:58.923388 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:58.965593 371 consensus_peers.cc:485] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:15:58.969247 30200 consensus_peers.cc:485] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:15:59.018445 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:59.031569 32709 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:15:59.078573 30069 consensus_peers.cc:485] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:15:59.103130 29809 consensus_peers.cc:485] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:15:59.110931 30070 consensus_peers.cc:485] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:59.173254 29809 consensus_peers.cc:485] T 7a6f23bb933949e9868139608456130f P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:15:59.182101 371 consensus_peers.cc:485] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:15:59.225741 29809 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:15:59.259382 30069 consensus_peers.cc:485] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:59.284525 371 consensus_peers.cc:485] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:15:59.308125 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:59.351588 29808 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:59.389982 30200 consensus_peers.cc:485] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:15:59.441864 30070 consensus_peers.cc:485] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:15:59.476096 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:59.487697 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:15:59.493991 30070 consensus_peers.cc:485] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:15:59.547683 29939 consensus_peers.cc:485] T 4563386631774e74a4b12fc71714c1a0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:59.654983 30070 consensus_peers.cc:485] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:15:59.684034 30070 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:15:59.684250 32709 consensus_peers.cc:485] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:15:59.701408 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:15:59.725852 30199 consensus_peers.cc:485] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:59.754702 30199 consensus_peers.cc:485] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:59.757148 32709 consensus_peers.cc:485] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:15:59.807534 30200 consensus_peers.cc:485] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:15:59.808528 29940 consensus_peers.cc:485] T 20e81486fbbe44e8888fb7c5bb5b355e P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:15:59.849148 32708 consensus_peers.cc:485] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:59.861542 30200 consensus_peers.cc:485] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:15:59.871094 30070 consensus_peers.cc:485] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:15:59.890816 29808 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:59.917408 29808 consensus_peers.cc:485] T 7527cc5931e24fa180efedbf81438efd P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:15:59.929020 29808 consensus_peers.cc:485] T 493c813acc27492fb952791cf2dc12a2 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:59.972461 30199 consensus_peers.cc:485] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:15:59.976163 32708 consensus_peers.cc:485] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:16:00.039426 371 consensus_peers.cc:485] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:16:00.047416 30199 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:16:00.053151 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:16:00.080979 371 consensus_peers.cc:485] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:16:00.081295 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:16:00.095048 30069 consensus_peers.cc:485] T e4fe8ae88ae843609a66c7a2e6c7b4af P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:16:00.120838 30200 consensus_peers.cc:485] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:16:00.136670 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:16:00.148578 30070 consensus_peers.cc:485] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:16:00.185972 30200 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:16:00.198907 30199 consensus_peers.cc:485] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:16:00.206923 29940 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:16:00.221804 32708 consensus_peers.cc:485] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:16:00.247215 32708 consensus_peers.cc:485] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:16:00.259610 32708 consensus_peers.cc:485] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 02:16:00.325464 29853 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:16:00.355350 29985 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:16:00.398190 32755 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 02:16:00.421878 370 consensus_peers.cc:485] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 02:16:00.433182 30244 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:16:00.437438 30114 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:16:00.476979 29808 consensus_peers.cc:485] T 46d608d84863424e8c65feac7c2e835f P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 02:16:00.479746 415 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 02:16:00.486243 30070 consensus_peers.cc:485] T 8fdbaf1df10b46d5a47326415d0cd0e9 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:16:00.504344 370 consensus_peers.cc:485] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:16:00.546116 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:16:00.641793 30069 consensus_peers.cc:485] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:16:00.655455 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:16:00.675289 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:16:00.702615 30199 consensus_peers.cc:485] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:16:00.719738 370 consensus_peers.cc:485] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:16:00.735450 370 consensus_peers.cc:485] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:16:00.743535 370 consensus_peers.cc:485] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:16:00.788376 29809 consensus_peers.cc:485] T 16cf978ac35341d4adea71a17a91194c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:16:00.799439 29809 consensus_peers.cc:485] T 9cd1418eb38d420c9c41d660476eafb0 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:16:00.828609 29940 consensus_peers.cc:485] T 0007679ffe8047d79dd8ac778fb6f1ce P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:16:00.832834 29809 consensus_peers.cc:485] T 0ec95509de3443bd96bf91a08ef7df4a P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:16:00.900940 32708 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:16:00.905018 29808 consensus_peers.cc:485] T 3533364f91c3433a8c1536b3eaee7424 P 16440e365430405fb10366ab70e2b429 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:16:00.913962 30070 consensus_peers.cc:485] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:16:00.974159 32708 consensus_peers.cc:485] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:16:01.007030 30069 consensus_peers.cc:485] T 779a6685fc8c4670a71b51c4493bc2f7 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:16:01.084213 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:16:01.098323 30199 consensus_peers.cc:485] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:16:01.110025 30069 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 02:16:01.138180 17439 raft_consensus.cc:482] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:16:01.138382 17439 raft_consensus.cc:504] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:16:01.139145 17439 leader_election.cc:288] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:16:01.143641 370 leader_election.cc:334] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:16:01.159694 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:16:01.162384 371 leader_election.cc:334] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:16:01.162523 371 leader_election.cc:302] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:16:01.162781 17439 raft_consensus.cc:2726] T ba97049157a646c780fa203d34cd4c82 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1203: Failure
Value of: s.ok()
Actual: false
Expected: true
Runtime error: /tmp/dist-test-task1Onrea/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 3
Maximum Replica Count | 40
Average Replica Count | 21.375000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 02:12:55.848443 624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:55.848785 624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:55.857342 624 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20240503 02:12:56.450268 698 meta_cache.cc:1015] Not found: LookupRpc { table: 'rebalancer_dynamic_table_new_name', partition-key: (<start>), attempt: 1 } failed: table rebalancer_dynamic_table_new_name was deleted: Table deleted at 2024-05-03 02:12:56 UTC
W20240503 02:12:56.470664 698 meta_cache.cc:788] Not found: LookupRpcById { tablet: '70820e67bc3740a08bcf0510a055c544', attempt: 1 } failed
I20240503 02:12:57.700474 624 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 02:12:58.206207 923 meta_cache.cc:1015] Not found: LookupRpc { table: 'rebalancer_dynamic_table_new_name', partition-key: (<start>), attempt: 1 } failed: table rebalancer_dynamic_table_new_name was deleted: Table deleted at 2024-05-03 02:12:58 UTC
W20240503 02:12:58.958019 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 60b67e8bde6c4177a72606ad2be64acf: could not find any suitable replica to move from server 155735c7f8b14997ac4c770aee2a9401 to server f575ce92c0224eeb80b9fae5f6561fbe
W20240503 02:12:58.958721 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 60b67e8bde6c4177a72606ad2be64acf: could not find any suitable replica to move from server 1f10f80f13c84e80b03eda7085a3b012 to server 8a666e798fe8477d8dd1e51b385d5ead
W20240503 02:12:58.958928 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 079f836c26424ee4adc122852f327cc7: could not find any suitable replica to move from server 1f10f80f13c84e80b03eda7085a3b012 to server 8a666e798fe8477d8dd1e51b385d5ead
W20240503 02:12:58.959160 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f6554714a9d84d2d864dca86f3ad3642: could not find any suitable replica to move from server 1f10f80f13c84e80b03eda7085a3b012 to server fbbc2be6382e47908ecb7c68c367bd26
W20240503 02:12:58.959241 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9498d0be60bf4564a9127c17a357f7ed: could not find any suitable replica to move from server 1f10f80f13c84e80b03eda7085a3b012 to server f575ce92c0224eeb80b9fae5f6561fbe
W20240503 02:12:58.959452 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 521f00c464174dec80c6db200c35e142: could not find any suitable replica to move from server 155735c7f8b14997ac4c770aee2a9401 to server 9a4d92bc25374fb3960fde7d10be309d
W20240503 02:12:58.959570 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 60b67e8bde6c4177a72606ad2be64acf: could not find any suitable replica to move from server 1f10f80f13c84e80b03eda7085a3b012 to server fbbc2be6382e47908ecb7c68c367bd26
W20240503 02:12:58.959738 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 99dbec2a2b63467ab34b96ecc8479244: could not find any suitable replica to move from server a9094240860e43bfb129193f3bfd5be6 to server fbbc2be6382e47908ecb7c68c367bd26
W20240503 02:12:58.959877 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e80de094521a4d71bc2fb5f3343275b8: could not find any suitable replica to move from server 1f10f80f13c84e80b03eda7085a3b012 to server f575ce92c0224eeb80b9fae5f6561fbe
W20240503 02:12:58.960004 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c501c7cdd0d142858bc57dffeae345f8: could not find any suitable replica to move from server a9094240860e43bfb129193f3bfd5be6 to server 9a4d92bc25374fb3960fde7d10be309d
W20240503 02:12:58.960140 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3c36e49b52c04307a655de366327db8c: could not find any suitable replica to move from server 1f10f80f13c84e80b03eda7085a3b012 to server fbbc2be6382e47908ecb7c68c367bd26
W20240503 02:12:58.960335 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c9cc90e5637140278312ab93695a8afd: could not find any suitable replica to move from server fbbc2be6382e47908ecb7c68c367bd26 to server 155735c7f8b14997ac4c770aee2a9401
W20240503 02:12:58.960423 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0cc2eea0038e4971a3d7119cbaabc735: could not find any suitable replica to move from server 1f10f80f13c84e80b03eda7085a3b012 to server fbbc2be6382e47908ecb7c68c367bd26
W20240503 02:12:58.960559 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 28d7eeaed8114076b7afb44e4f89a2ea: could not find any suitable replica to move from server a9094240860e43bfb129193f3bfd5be6 to server 8a666e798fe8477d8dd1e51b385d5ead
W20240503 02:12:58.960642 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table df07c1fd4e654fea9a59ea9a0ea79595: could not find any suitable replica to move from server 155735c7f8b14997ac4c770aee2a9401 to server f575ce92c0224eeb80b9fae5f6561fbe
W20240503 02:12:58.960704 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 60b67e8bde6c4177a72606ad2be64acf: could not find any suitable replica to move from server 155735c7f8b14997ac4c770aee2a9401 to server 9a4d92bc25374fb3960fde7d10be309d
W20240503 02:12:58.960796 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6fdcf94cbd1844948615de6c2a4d18cc: could not find any suitable replica to move from server 1f10f80f13c84e80b03eda7085a3b012 to server fbbc2be6382e47908ecb7c68c367bd26
W20240503 02:12:58.960934 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3131069dba4a4f9aa26d4c134fcb4704: could not find any suitable replica to move from server a9094240860e43bfb129193f3bfd5be6 to server 9a4d92bc25374fb3960fde7d10be309d
W20240503 02:12:58.961097 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6bbf46811ff140d289e600493149f118: could not find any suitable replica to move from server 155735c7f8b14997ac4c770aee2a9401 to server f575ce92c0224eeb80b9fae5f6561fbe
W20240503 02:12:58.961166 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 28d7eeaed8114076b7afb44e4f89a2ea: could not find any suitable replica to move from server 1f10f80f13c84e80b03eda7085a3b012 to server 8a666e798fe8477d8dd1e51b385d5ead
W20240503 02:12:58.961227 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 521f00c464174dec80c6db200c35e142: could not find any suitable replica to move from server 1f10f80f13c84e80b03eda7085a3b012 to server fbbc2be6382e47908ecb7c68c367bd26
W20240503 02:12:58.961285 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 60b67e8bde6c4177a72606ad2be64acf: could not find any suitable replica to move from server 16440e365430405fb10366ab70e2b429 to server 8a666e798fe8477d8dd1e51b385d5ead
W20240503 02:12:58.961340 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table df07c1fd4e654fea9a59ea9a0ea79595: could not find any suitable replica to move from server 1f10f80f13c84e80b03eda7085a3b012 to server fbbc2be6382e47908ecb7c68c367bd26
W20240503 02:12:58.961421 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 99dbec2a2b63467ab34b96ecc8479244: could not find any suitable replica to move from server 155735c7f8b14997ac4c770aee2a9401 to server f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:12:59.056231 624 rebalancer_tool.cc:1074] tablet a592cdf6d1d747c68b5334b90006fa8b: '16440e365430405fb10366ab70e2b429' -> '8a666e798fe8477d8dd1e51b385d5ead' move scheduled
I20240503 02:12:59.274120 624 rebalancer_tool.cc:1074] tablet 35a5a7f4fe184b9fadc94c98b61deaf0: 'a9094240860e43bfb129193f3bfd5be6' -> 'fbbc2be6382e47908ecb7c68c367bd26' move scheduled
I20240503 02:12:59.275671 624 rebalancer_tool.cc:1084] tablet a74d38f7d6e5441396944c52acd41c74: '1f10f80f13c84e80b03eda7085a3b012' -> 'f575ce92c0224eeb80b9fae5f6561fbe' move ignored: Not found: a74d38f7d6e5441396944c52acd41c74: tablet not found
I20240503 02:12:59.275856 624 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:13:00.322676 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 079f836c26424ee4adc122852f327cc7: could not find any suitable replica to move from server 1f10f80f13c84e80b03eda7085a3b012 to server 9a4d92bc25374fb3960fde7d10be309d
W20240503 02:13:00.323050 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 079f836c26424ee4adc122852f327cc7: could not find any suitable replica to move from server 16440e365430405fb10366ab70e2b429 to server 8a666e798fe8477d8dd1e51b385d5ead
W20240503 02:13:00.323405 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6fdcf94cbd1844948615de6c2a4d18cc: could not find any suitable replica to move from server 16440e365430405fb10366ab70e2b429 to server 9a4d92bc25374fb3960fde7d10be309d
W20240503 02:13:00.323511 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 28d7eeaed8114076b7afb44e4f89a2ea: could not find any suitable replica to move from server 1f10f80f13c84e80b03eda7085a3b012 to server 9a4d92bc25374fb3960fde7d10be309d
W20240503 02:13:00.323613 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 27940b10e50e4a418d9e7155f5a25394: could not find any suitable replica to move from server 8a666e798fe8477d8dd1e51b385d5ead to server 16440e365430405fb10366ab70e2b429
W20240503 02:13:00.323858 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 079f836c26424ee4adc122852f327cc7: could not find any suitable replica to move from server 16440e365430405fb10366ab70e2b429 to server f575ce92c0224eeb80b9fae5f6561fbe
W20240503 02:13:00.323992 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 53af41732fcb40aea2f1c6f0ac294bde: could not find any suitable replica to move from server f575ce92c0224eeb80b9fae5f6561fbe to server 9a4d92bc25374fb3960fde7d10be309d
W20240503 02:13:00.324251 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3c36e49b52c04307a655de366327db8c: could not find any suitable replica to move from server a9094240860e43bfb129193f3bfd5be6 to server 9a4d92bc25374fb3960fde7d10be309d
W20240503 02:13:00.324394 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0cc2eea0038e4971a3d7119cbaabc735: could not find any suitable replica to move from server a9094240860e43bfb129193f3bfd5be6 to server fbbc2be6382e47908ecb7c68c367bd26
W20240503 02:13:00.324537 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f6554714a9d84d2d864dca86f3ad3642: could not find any suitable replica to move from server 1f10f80f13c84e80b03eda7085a3b012 to server 9a4d92bc25374fb3960fde7d10be309d
W20240503 02:13:00.324631 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 521f00c464174dec80c6db200c35e142: could not find any suitable replica to move from server 155735c7f8b14997ac4c770aee2a9401 to server f575ce92c0224eeb80b9fae5f6561fbe
W20240503 02:13:00.324735 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e80de094521a4d71bc2fb5f3343275b8: could not find any suitable replica to move from server 1f10f80f13c84e80b03eda7085a3b012 to server 8a666e798fe8477d8dd1e51b385d5ead
W20240503 02:13:00.324930 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 99dbec2a2b63467ab34b96ecc8479244: could not find any suitable replica to move from server 1f10f80f13c84e80b03eda7085a3b012 to server fbbc2be6382e47908ecb7c68c367bd26
W20240503 02:13:00.325075 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table df07c1fd4e654fea9a59ea9a0ea79595: could not find any suitable replica to move from server 155735c7f8b14997ac4c770aee2a9401 to server fbbc2be6382e47908ecb7c68c367bd26
W20240503 02:13:00.325177 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 079f836c26424ee4adc122852f327cc7: could not find any suitable replica to move from server 1f10f80f13c84e80b03eda7085a3b012 to server fbbc2be6382e47908ecb7c68c367bd26
W20240503 02:13:00.325307 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3131069dba4a4f9aa26d4c134fcb4704: could not find any suitable replica to move from server a9094240860e43bfb129193f3bfd5be6 to server 9a4d92bc25374fb3960fde7d10be309d
W20240503 02:13:00.326476 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9498d0be60bf4564a9127c17a357f7ed: could not find any suitable replica to move from server 155735c7f8b14997ac4c770aee2a9401 to server f575ce92c0224eeb80b9fae5f6561fbe
W20240503 02:13:00.327558 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6bbf46811ff140d289e600493149f118: could not find any suitable replica to move from server a9094240860e43bfb129193f3bfd5be6 to server f575ce92c0224eeb80b9fae5f6561fbe
W20240503 02:13:00.327682 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4c683f284ec140f69703e915a553a141: could not find any suitable replica to move from server 155735c7f8b14997ac4c770aee2a9401 to server 8a666e798fe8477d8dd1e51b385d5ead
W20240503 02:13:00.327800 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 28d7eeaed8114076b7afb44e4f89a2ea: could not find any suitable replica to move from server a9094240860e43bfb129193f3bfd5be6 to server 9a4d92bc25374fb3960fde7d10be309d
W20240503 02:13:00.327878 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 53af41732fcb40aea2f1c6f0ac294bde: could not find any suitable replica to move from server fbbc2be6382e47908ecb7c68c367bd26 to server 8a666e798fe8477d8dd1e51b385d5ead
I20240503 02:13:00.399925 624 rebalancer_tool.cc:1074] tablet 0ae1063008d44f6eaa61a6e353732e48: '155735c7f8b14997ac4c770aee2a9401' -> '9a4d92bc25374fb3960fde7d10be309d' move scheduled
I20240503 02:13:00.495870 624 rebalancer_tool.cc:1074] tablet 83c7d1060ccf49a2adda4302600a9876: 'f575ce92c0224eeb80b9fae5f6561fbe' -> '1f10f80f13c84e80b03eda7085a3b012' move scheduled
I20240503 02:13:00.585307 624 rebalancer_tool.cc:1074] tablet 5f4b72b138c64ac58e6477cd2b1fb215: 'a9094240860e43bfb129193f3bfd5be6' -> '8a666e798fe8477d8dd1e51b385d5ead' move scheduled
I20240503 02:13:00.637375 624 rebalancer_tool.cc:1074] tablet 96de1c26ae884ea18b63b328db040ba1: '16440e365430405fb10366ab70e2b429' -> 'fbbc2be6382e47908ecb7c68c367bd26' move scheduled
I20240503 02:13:00.716262 624 rebalancer_tool.cc:1074] tablet 8fdbaf1df10b46d5a47326415d0cd0e9: '155735c7f8b14997ac4c770aee2a9401' -> '9a4d92bc25374fb3960fde7d10be309d' move scheduled
I20240503 02:13:00.806500 624 rebalancer_tool.cc:1074] tablet 7527cc5931e24fa180efedbf81438efd: '1f10f80f13c84e80b03eda7085a3b012' -> 'f575ce92c0224eeb80b9fae5f6561fbe' move scheduled
I20240503 02:13:00.970486 624 rebalancer_tool.cc:1074] tablet 08ee37b160d441ddabb640be907d6771: 'a9094240860e43bfb129193f3bfd5be6' -> '8a666e798fe8477d8dd1e51b385d5ead' move scheduled
I20240503 02:13:01.080153 624 rebalancer_tool.cc:1074] tablet 7076453cbe5944eb9b075127e1e0495d: '16440e365430405fb10366ab70e2b429' -> '9a4d92bc25374fb3960fde7d10be309d' move scheduled
I20240503 02:13:01.417519 624 rebalancer_tool.cc:1074] tablet b6c9fd3cbbd5453897aa072600db569d: '155735c7f8b14997ac4c770aee2a9401' -> 'fbbc2be6382e47908ecb7c68c367bd26' move scheduled
I20240503 02:13:01.420758 624 rebalancer_tool.cc:1084] tablet 4de7cb0be278476481d01083add5c978: '1f10f80f13c84e80b03eda7085a3b012' -> 'f575ce92c0224eeb80b9fae5f6561fbe' move ignored: Not found: 4de7cb0be278476481d01083add5c978: tablet not found
I20240503 02:13:01.420897 624 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:13:02.550418 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3131069dba4a4f9aa26d4c134fcb4704: could not find any suitable replica to move from server 16440e365430405fb10366ab70e2b429 to server 8a666e798fe8477d8dd1e51b385d5ead
W20240503 02:13:02.550658 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 28d7eeaed8114076b7afb44e4f89a2ea: could not find any suitable replica to move from server a9094240860e43bfb129193f3bfd5be6 to server 9a4d92bc25374fb3960fde7d10be309d
W20240503 02:13:02.550873 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table df07c1fd4e654fea9a59ea9a0ea79595: could not find any suitable replica to move from server 155735c7f8b14997ac4c770aee2a9401 to server f575ce92c0224eeb80b9fae5f6561fbe
W20240503 02:13:02.550969 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9498d0be60bf4564a9127c17a357f7ed: could not find any suitable replica to move from server 155735c7f8b14997ac4c770aee2a9401 to server fbbc2be6382e47908ecb7c68c367bd26
W20240503 02:13:02.551087 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3c36e49b52c04307a655de366327db8c: could not find any suitable replica to move from server a9094240860e43bfb129193f3bfd5be6 to server f575ce92c0224eeb80b9fae5f6561fbe
W20240503 02:13:02.551213 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c14bbf6b6bfd41148c82b54717aceb15: could not find any suitable replica to move from server fbbc2be6382e47908ecb7c68c367bd26 to server 1f10f80f13c84e80b03eda7085a3b012
W20240503 02:13:02.551280 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f6554714a9d84d2d864dca86f3ad3642: could not find any suitable replica to move from server 1f10f80f13c84e80b03eda7085a3b012 to server 8a666e798fe8477d8dd1e51b385d5ead
W20240503 02:13:02.551385 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e80de094521a4d71bc2fb5f3343275b8: could not find any suitable replica to move from server 1f10f80f13c84e80b03eda7085a3b012 to server 8a666e798fe8477d8dd1e51b385d5ead
W20240503 02:13:02.551465 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 99dbec2a2b63467ab34b96ecc8479244: could not find any suitable replica to move from server a9094240860e43bfb129193f3bfd5be6 to server fbbc2be6382e47908ecb7c68c367bd26
W20240503 02:13:02.551533 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6fdcf94cbd1844948615de6c2a4d18cc: could not find any suitable replica to move from server 16440e365430405fb10366ab70e2b429 to server fbbc2be6382e47908ecb7c68c367bd26
W20240503 02:13:02.551652 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6bbf46811ff140d289e600493149f118: could not find any suitable replica to move from server a9094240860e43bfb129193f3bfd5be6 to server 8a666e798fe8477d8dd1e51b385d5ead
W20240503 02:13:02.551712 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c985f9c1d7a040d793cf0eaab4c1c6c5: could not find any suitable replica to move from server fbbc2be6382e47908ecb7c68c367bd26 to server 9a4d92bc25374fb3960fde7d10be309d
W20240503 02:13:02.551780 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c9cc90e5637140278312ab93695a8afd: could not find any suitable replica to move from server fbbc2be6382e47908ecb7c68c367bd26 to server f575ce92c0224eeb80b9fae5f6561fbe
W20240503 02:13:02.551852 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 27940b10e50e4a418d9e7155f5a25394: could not find any suitable replica to move from server 8a666e798fe8477d8dd1e51b385d5ead to server 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:13:02.602283 624 rebalancer_tool.cc:1074] tablet 88b57085de3144cea3f64922ae5ccf55: '1f10f80f13c84e80b03eda7085a3b012' -> 'f575ce92c0224eeb80b9fae5f6561fbe' move scheduled
I20240503 02:13:02.704862 624 rebalancer_tool.cc:1074] tablet 6bd4fcc4faf44d148a03beaafc1c05ba: '8a666e798fe8477d8dd1e51b385d5ead' -> '9a4d92bc25374fb3960fde7d10be309d' move scheduled
I20240503 02:13:02.755481 624 rebalancer_tool.cc:1074] tablet df37d88053e2419fbe40d03023d0c756: 'a9094240860e43bfb129193f3bfd5be6' -> 'fbbc2be6382e47908ecb7c68c367bd26' move scheduled
I20240503 02:13:02.827178 624 rebalancer_tool.cc:1074] tablet 779a6685fc8c4670a71b51c4493bc2f7: '16440e365430405fb10366ab70e2b429' -> 'f575ce92c0224eeb80b9fae5f6561fbe' move scheduled
I20240503 02:13:02.968943 624 rebalancer_tool.cc:1074] tablet 6bf84757d1fb44689da830a82e857787: '155735c7f8b14997ac4c770aee2a9401' -> '9a4d92bc25374fb3960fde7d10be309d' move scheduled
I20240503 02:13:03.014794 624 rebalancer_tool.cc:1074] tablet 32a2dbf80daf40d6b21442a39f62f14c: '1f10f80f13c84e80b03eda7085a3b012' -> '8a666e798fe8477d8dd1e51b385d5ead' move scheduled
I20240503 02:13:03.071693 624 rebalancer_tool.cc:1074] tablet e61579e74a1a4a9dbd9a09f9b34d9c8d: 'a9094240860e43bfb129193f3bfd5be6' -> 'fbbc2be6382e47908ecb7c68c367bd26' move scheduled
I20240503 02:13:03.345593 624 rebalancer_tool.cc:1074] tablet 1c9200fa969a46e6a48c64293801382b: '16440e365430405fb10366ab70e2b429' -> 'f575ce92c0224eeb80b9fae5f6561fbe' move scheduled
I20240503 02:13:08.564431 624 rebalancer_tool.cc:1123] tablet a592cdf6d1d747c68b5334b90006fa8b: '16440e365430405fb10366ab70e2b429' -> '8a666e798fe8477d8dd1e51b385d5ead' move is abandoned: Not found: a592cdf6d1d747c68b5334b90006fa8b: tablet not found
E20240503 02:13:12.451104 2842 ksck_remote.cc:634] unable to open table rebalancer_dynamic_table_new_name: Not found: the table does not exist: table_name: "rebalancer_dynamic_table_new_name"
I20240503 02:13:17.339207 624 rebalancer_tool.cc:1123] tablet 0ae1063008d44f6eaa61a6e353732e48: '155735c7f8b14997ac4c770aee2a9401' -> '9a4d92bc25374fb3960fde7d10be309d' move is abandoned: Not found: 0ae1063008d44f6eaa61a6e353732e48: tablet not found
I20240503 02:13:21.711186 624 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:13:22.874245 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 854d24d23fe34d699e6656ef51d15693: could not find any suitable replica to move from server 1f10f80f13c84e80b03eda7085a3b012 to server 155735c7f8b14997ac4c770aee2a9401
W20240503 02:13:22.874444 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e4b9d13872ad4d27841df71e38d5482f: could not find any suitable replica to move from server a9094240860e43bfb129193f3bfd5be6 to server fbbc2be6382e47908ecb7c68c367bd26
W20240503 02:13:22.874788 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 99dbec2a2b63467ab34b96ecc8479244: could not find any suitable replica to move from server a9094240860e43bfb129193f3bfd5be6 to server fbbc2be6382e47908ecb7c68c367bd26
W20240503 02:13:22.874904 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6bbf46811ff140d289e600493149f118: could not find any suitable replica to move from server 1f10f80f13c84e80b03eda7085a3b012 to server 9a4d92bc25374fb3960fde7d10be309d
W20240503 02:13:22.875025 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6fdcf94cbd1844948615de6c2a4d18cc: could not find any suitable replica to move from server 16440e365430405fb10366ab70e2b429 to server fbbc2be6382e47908ecb7c68c367bd26
W20240503 02:13:22.875161 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff2eb02eb88b44a0b0da2abc6307220a: could not find any suitable replica to move from server 8a666e798fe8477d8dd1e51b385d5ead to server f575ce92c0224eeb80b9fae5f6561fbe
W20240503 02:13:22.875306 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c14bbf6b6bfd41148c82b54717aceb15: could not find any suitable replica to move from server fbbc2be6382e47908ecb7c68c367bd26 to server 1f10f80f13c84e80b03eda7085a3b012
W20240503 02:13:22.875524 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c9cc90e5637140278312ab93695a8afd: could not find any suitable replica to move from server fbbc2be6382e47908ecb7c68c367bd26 to server 155735c7f8b14997ac4c770aee2a9401
W20240503 02:13:22.875695 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9498d0be60bf4564a9127c17a357f7ed: could not find any suitable replica to move from server 155735c7f8b14997ac4c770aee2a9401 to server 9a4d92bc25374fb3960fde7d10be309d
W20240503 02:13:22.875839 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e80de094521a4d71bc2fb5f3343275b8: could not find any suitable replica to move from server 16440e365430405fb10366ab70e2b429 to server 8a666e798fe8477d8dd1e51b385d5ead
W20240503 02:13:22.875955 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table df07c1fd4e654fea9a59ea9a0ea79595: could not find any suitable replica to move from server 1f10f80f13c84e80b03eda7085a3b012 to server f575ce92c0224eeb80b9fae5f6561fbe
W20240503 02:13:22.876120 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3131069dba4a4f9aa26d4c134fcb4704: could not find any suitable replica to move from server 155735c7f8b14997ac4c770aee2a9401 to server 9a4d92bc25374fb3960fde7d10be309d
W20240503 02:13:22.876240 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3c36e49b52c04307a655de366327db8c: could not find any suitable replica to move from server 155735c7f8b14997ac4c770aee2a9401 to server fbbc2be6382e47908ecb7c68c367bd26
W20240503 02:13:22.876354 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 95e60300d67c446987cfb1f3dbeecf3a: could not find any suitable replica to move from server 1f10f80f13c84e80b03eda7085a3b012 to server a9094240860e43bfb129193f3bfd5be6
W20240503 02:13:22.876608 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d70cb2e6817a47df8925321a6a778740: could not find any suitable replica to move from server 8a666e798fe8477d8dd1e51b385d5ead to server a9094240860e43bfb129193f3bfd5be6
W20240503 02:13:22.878494 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 28d7eeaed8114076b7afb44e4f89a2ea: could not find any suitable replica to move from server a9094240860e43bfb129193f3bfd5be6 to server 8a666e798fe8477d8dd1e51b385d5ead
W20240503 02:13:22.878736 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 196b7559fe174e1b963b8d38f2240e48: could not find any suitable replica to move from server a9094240860e43bfb129193f3bfd5be6 to server 1f10f80f13c84e80b03eda7085a3b012
W20240503 02:13:22.878886 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f6554714a9d84d2d864dca86f3ad3642: could not find any suitable replica to move from server 16440e365430405fb10366ab70e2b429 to server 8a666e798fe8477d8dd1e51b385d5ead
W20240503 02:13:22.879196 624 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 098275b181bb4441990148dfd09ded67: could not find any suitable replica to move from server f575ce92c0224eeb80b9fae5f6561fbe to server 1f10f80f13c84e80b03eda7085a3b012
I20240503 02:13:23.065663 624 rebalancer_tool.cc:1074] tablet 493c813acc27492fb952791cf2dc12a2: '155735c7f8b14997ac4c770aee2a9401' -> '9a4d92bc25374fb3960fde7d10be309d' move scheduled
I20240503 02:13:23.231791 624 rebalancer_tool.cc:1074] tablet 58e1aaecf47944569296fc1e0780efb5: '16440e365430405fb10366ab70e2b429' -> '1f10f80f13c84e80b03eda7085a3b012' move scheduled
I20240503 02:13:23.286275 624 rebalancer_tool.cc:1074] tablet e4fe8ae88ae843609a66c7a2e6c7b4af: '8a666e798fe8477d8dd1e51b385d5ead' -> '155735c7f8b14997ac4c770aee2a9401' move scheduled
I20240503 02:13:25.794605 624 rebalancer_tool.cc:1136] tablet 1c9200fa969a46e6a48c64293801382b: '16440e365430405fb10366ab70e2b429' -> 'f575ce92c0224eeb80b9fae5f6561fbe' move completed: OK
I20240503 02:13:25.804800 624 rebalancer_tool.cc:1136] tablet e61579e74a1a4a9dbd9a09f9b34d9c8d: 'a9094240860e43bfb129193f3bfd5be6' -> 'fbbc2be6382e47908ecb7c68c367bd26' move completed: OK
I20240503 02:13:25.814616 624 rebalancer_tool.cc:1136] tablet 96de1c26ae884ea18b63b328db040ba1: '16440e365430405fb10366ab70e2b429' -> 'fbbc2be6382e47908ecb7c68c367bd26' move completed: OK
I20240503 02:13:25.830078 624 rebalancer_tool.cc:1136] tablet 88b57085de3144cea3f64922ae5ccf55: '1f10f80f13c84e80b03eda7085a3b012' -> 'f575ce92c0224eeb80b9fae5f6561fbe' move completed: OK
I20240503 02:13:25.840869 624 rebalancer_tool.cc:1136] tablet 6bf84757d1fb44689da830a82e857787: '155735c7f8b14997ac4c770aee2a9401' -> '9a4d92bc25374fb3960fde7d10be309d' move completed: OK
I20240503 02:13:25.853960 624 rebalancer_tool.cc:1136] tablet 83c7d1060ccf49a2adda4302600a9876: 'f575ce92c0224eeb80b9fae5f6561fbe' -> '1f10f80f13c84e80b03eda7085a3b012' move completed: OK
I20240503 02:13:25.864125 624 rebalancer_tool.cc:1136] tablet 32a2dbf80daf40d6b21442a39f62f14c: '1f10f80f13c84e80b03eda7085a3b012' -> '8a666e798fe8477d8dd1e51b385d5ead' move completed: OK
I20240503 02:13:25.914994 624 rebalancer_tool.cc:1136] tablet 5f4b72b138c64ac58e6477cd2b1fb215: 'a9094240860e43bfb129193f3bfd5be6' -> '8a666e798fe8477d8dd1e51b385d5ead' move completed: OK
I20240503 02:13:26.739868 624 rebalancer_tool.cc:1136] tablet 35a5a7f4fe184b9fadc94c98b61deaf0: 'a9094240860e43bfb129193f3bfd5be6' -> 'fbbc2be6382e47908ecb7c68c367bd26' move completed: OK
I20240503 02:13:27.502105 624 rebalancer_tool.cc:1136] tablet 7076453cbe5944eb9b075127e1e0495d: '16440e365430405fb10366ab70e2b429' -> '9a4d92bc25374fb3960fde7d10be309d' move completed: OK
I20240503 02:13:27.529124 624 rebalancer_tool.cc:1136] tablet 08ee37b160d441ddabb640be907d6771: 'a9094240860e43bfb129193f3bfd5be6' -> '8a666e798fe8477d8dd1e51b385d5ead' move completed: OK
I20240503 02:13:28.177175 624 rebalancer_tool.cc:1136] tablet df37d88053e2419fbe40d03023d0c756: 'a9094240860e43bfb129193f3bfd5be6' -> 'fbbc2be6382e47908ecb7c68c367bd26' move completed: OK
I20240503 02:13:28.817322 624 rebalancer_tool.cc:1136] tablet b6c9fd3cbbd5453897aa072600db569d: '155735c7f8b14997ac4c770aee2a9401' -> 'fbbc2be6382e47908ecb7c68c367bd26' move completed: OK
I20240503 02:13:28.838897 624 rebalancer_tool.cc:1136] tablet 6bd4fcc4faf44d148a03beaafc1c05ba: '8a666e798fe8477d8dd1e51b385d5ead' -> '9a4d92bc25374fb3960fde7d10be309d' move completed: OK
I20240503 02:13:28.865343 624 rebalancer_tool.cc:1136] tablet 779a6685fc8c4670a71b51c4493bc2f7: '16440e365430405fb10366ab70e2b429' -> 'f575ce92c0224eeb80b9fae5f6561fbe' move completed: OK
I20240503 02:13:28.945147 624 rebalancer_tool.cc:1074] tablet 63b416f4e004490b8e38a96fd065dfea: 'a9094240860e43bfb129193f3bfd5be6' -> 'fbbc2be6382e47908ecb7c68c367bd26' move scheduled
I20240503 02:13:29.020092 624 rebalancer_tool.cc:1074] tablet 0ec95509de3443bd96bf91a08ef7df4a: '8a666e798fe8477d8dd1e51b385d5ead' -> '16440e365430405fb10366ab70e2b429' move scheduled
I20240503 02:13:29.073809 624 rebalancer_tool.cc:1074] tablet 70b8ebcf6f3245e6bd93eeb38ca1cbf5: 'f575ce92c0224eeb80b9fae5f6561fbe' -> 'a9094240860e43bfb129193f3bfd5be6' move scheduled
I20240503 02:13:29.133195 624 rebalancer_tool.cc:1074] tablet 1758ec3783b24d3c820ed0d6dc118e69: 'fbbc2be6382e47908ecb7c68c367bd26' -> '1f10f80f13c84e80b03eda7085a3b012' move scheduled
I20240503 02:13:29.205893 624 rebalancer_tool.cc:1074] tablet ec3891bfc4014b63afef6c259de00ca7: '9a4d92bc25374fb3960fde7d10be309d' -> '16440e365430405fb10366ab70e2b429' move scheduled
I20240503 02:13:29.260138 624 rebalancer_tool.cc:1074] tablet f694fefb42d84318a6d91532105b3c90: '8a666e798fe8477d8dd1e51b385d5ead' -> 'f575ce92c0224eeb80b9fae5f6561fbe' move scheduled
I20240503 02:13:29.311764 624 rebalancer_tool.cc:1074] tablet 04e47da8cf7445dcacaf1f7c8950bd34: 'a9094240860e43bfb129193f3bfd5be6' -> 'fbbc2be6382e47908ecb7c68c367bd26' move scheduled
I20240503 02:13:29.386870 624 rebalancer_tool.cc:1074] tablet 00aedc4902a745d6a859cbbf99529a7f: '9a4d92bc25374fb3960fde7d10be309d' -> '155735c7f8b14997ac4c770aee2a9401' move scheduled
I20240503 02:13:29.687367 624 rebalancer_tool.cc:1074] tablet 5040203bd48549ff9c8f64c71d6bc598: '1f10f80f13c84e80b03eda7085a3b012' -> 'f575ce92c0224eeb80b9fae5f6561fbe' move scheduled
I20240503 02:13:29.813441 624 rebalancer_tool.cc:1074] tablet 89938d4c793a44408795398078cc2ee0: '16440e365430405fb10366ab70e2b429' -> '8a666e798fe8477d8dd1e51b385d5ead' move scheduled
I20240503 02:13:29.898063 624 rebalancer_tool.cc:1074] tablet 9102ab9545c54e82b642673795cb747a: 'fbbc2be6382e47908ecb7c68c367bd26' -> 'a9094240860e43bfb129193f3bfd5be6' move scheduled
I20240503 02:13:30.062167 624 rebalancer_tool.cc:1074] tablet 83e8dad806564b128d5f0fe559d8ffe1: '9a4d92bc25374fb3960fde7d10be309d' -> '155735c7f8b14997ac4c770aee2a9401' move scheduled
I20240503 02:13:30.130488 624 rebalancer_tool.cc:1074] tablet e165c71f8b6645a6936b58b2a8da1842: 'f575ce92c0224eeb80b9fae5f6561fbe' -> '1f10f80f13c84e80b03eda7085a3b012' move scheduled
I20240503 02:13:30.275415 624 rebalancer_tool.cc:1074] tablet 16cf978ac35341d4adea71a17a91194c: '8a666e798fe8477d8dd1e51b385d5ead' -> '16440e365430405fb10366ab70e2b429' move scheduled
I20240503 02:13:30.363061 624 rebalancer_tool.cc:1074] tablet 9fa3f1e452c24c7d83a36f10284b8b5d: 'fbbc2be6382e47908ecb7c68c367bd26' -> 'a9094240860e43bfb129193f3bfd5be6' move scheduled
I20240503 02:13:39.717046 624 rebalancer_tool.cc:1136] tablet 8fdbaf1df10b46d5a47326415d0cd0e9: '155735c7f8b14997ac4c770aee2a9401' -> '9a4d92bc25374fb3960fde7d10be309d' move completed: OK
I20240503 02:13:39.727070 624 rebalancer_tool.cc:1136] tablet 7527cc5931e24fa180efedbf81438efd: '1f10f80f13c84e80b03eda7085a3b012' -> 'f575ce92c0224eeb80b9fae5f6561fbe' move completed: OK
I20240503 02:13:40.537973 624 rebalancer_tool.cc:1136] tablet 493c813acc27492fb952791cf2dc12a2: '155735c7f8b14997ac4c770aee2a9401' -> '9a4d92bc25374fb3960fde7d10be309d' move completed: OK
I20240503 02:13:42.348366 624 rebalancer_tool.cc:1136] tablet e4fe8ae88ae843609a66c7a2e6c7b4af: '8a666e798fe8477d8dd1e51b385d5ead' -> '155735c7f8b14997ac4c770aee2a9401' move completed: OK
I20240503 02:13:42.471272 624 rebalancer_tool.cc:1074] tablet c87e7cbed0e843f5b5bb9bddef598cdb: 'f575ce92c0224eeb80b9fae5f6561fbe' -> '155735c7f8b14997ac4c770aee2a9401' move scheduled
I20240503 02:13:42.557682 624 rebalancer_tool.cc:1074] tablet 5800022ae50043ee8b1af1e134e86e6a: '9a4d92bc25374fb3960fde7d10be309d' -> '1f10f80f13c84e80b03eda7085a3b012' move scheduled
I20240503 02:13:42.617378 624 rebalancer_tool.cc:1074] tablet db0756f0f331402aaef0583e1f9e5318: '155735c7f8b14997ac4c770aee2a9401' -> '8a666e798fe8477d8dd1e51b385d5ead' move scheduled
I20240503 02:13:45.761343 624 rebalancer_tool.cc:1136] tablet e165c71f8b6645a6936b58b2a8da1842: 'f575ce92c0224eeb80b9fae5f6561fbe' -> '1f10f80f13c84e80b03eda7085a3b012' move completed: OK
I20240503 02:13:45.785370 624 rebalancer_tool.cc:1136] tablet 83e8dad806564b128d5f0fe559d8ffe1: '9a4d92bc25374fb3960fde7d10be309d' -> '155735c7f8b14997ac4c770aee2a9401' move completed: OK
I20240503 02:13:45.795765 624 rebalancer_tool.cc:1136] tablet 5040203bd48549ff9c8f64c71d6bc598: '1f10f80f13c84e80b03eda7085a3b012' -> 'f575ce92c0224eeb80b9fae5f6561fbe' move completed: OK
I20240503 02:13:45.805418 624 rebalancer_tool.cc:1136] tablet 00aedc4902a745d6a859cbbf99529a7f: '9a4d92bc25374fb3960fde7d10be309d' -> '155735c7f8b14997ac4c770aee2a9401' move completed: OK
I20240503 02:13:45.825096 624 rebalancer_tool.cc:1136] tablet ec3891bfc4014b63afef6c259de00ca7: '9a4d92bc25374fb3960fde7d10be309d' -> '16440e365430405fb10366ab70e2b429' move completed: OK
I20240503 02:13:45.839346 624 rebalancer_tool.cc:1136] tablet 1758ec3783b24d3c820ed0d6dc118e69: 'fbbc2be6382e47908ecb7c68c367bd26' -> '1f10f80f13c84e80b03eda7085a3b012' move completed: OK
I20240503 02:13:45.850833 624 rebalancer_tool.cc:1136] tablet 9fa3f1e452c24c7d83a36f10284b8b5d: 'fbbc2be6382e47908ecb7c68c367bd26' -> 'a9094240860e43bfb129193f3bfd5be6' move completed: OK
I20240503 02:13:45.859668 624 rebalancer_tool.cc:1136] tablet 70b8ebcf6f3245e6bd93eeb38ca1cbf5: 'f575ce92c0224eeb80b9fae5f6561fbe' -> 'a9094240860e43bfb129193f3bfd5be6' move completed: OK
I20240503 02:13:45.890391 624 rebalancer_tool.cc:1136] tablet 0ec95509de3443bd96bf91a08ef7df4a: '8a666e798fe8477d8dd1e51b385d5ead' -> '16440e365430405fb10366ab70e2b429' move completed: OK
I20240503 02:13:45.907800 624 rebalancer_tool.cc:1136] tablet 63b416f4e004490b8e38a96fd065dfea: 'a9094240860e43bfb129193f3bfd5be6' -> 'fbbc2be6382e47908ecb7c68c367bd26' move completed: OK
I20240503 02:13:45.920374 624 rebalancer_tool.cc:1136] tablet 58e1aaecf47944569296fc1e0780efb5: '16440e365430405fb10366ab70e2b429' -> '1f10f80f13c84e80b03eda7085a3b012' move completed: OK
I20240503 02:13:46.617491 624 rebalancer_tool.cc:1136] tablet 9102ab9545c54e82b642673795cb747a: 'fbbc2be6382e47908ecb7c68c367bd26' -> 'a9094240860e43bfb129193f3bfd5be6' move completed: OK
I20240503 02:13:46.629256 624 rebalancer_tool.cc:1136] tablet 04e47da8cf7445dcacaf1f7c8950bd34: 'a9094240860e43bfb129193f3bfd5be6' -> 'fbbc2be6382e47908ecb7c68c367bd26' move completed: OK
I20240503 02:13:47.291168 624 rebalancer_tool.cc:1074] tablet c2d33db64ee8482aad840af219d60b8b: 'fbbc2be6382e47908ecb7c68c367bd26' -> 'a9094240860e43bfb129193f3bfd5be6' move scheduled
I20240503 02:13:47.349951 624 rebalancer_tool.cc:1074] tablet 4cc7861a21f94dabb01133bac9ba0b46: '9a4d92bc25374fb3960fde7d10be309d' -> '1f10f80f13c84e80b03eda7085a3b012' move scheduled
I20240503 02:13:47.487452 624 rebalancer_tool.cc:1074] tablet ac4d497de05c49edbd6e0edac73c058d: 'fbbc2be6382e47908ecb7c68c367bd26' -> '155735c7f8b14997ac4c770aee2a9401' move scheduled
I20240503 02:13:47.590214 624 rebalancer_tool.cc:1074] tablet 46d608d84863424e8c65feac7c2e835f: 'a9094240860e43bfb129193f3bfd5be6' -> 'f575ce92c0224eeb80b9fae5f6561fbe' move scheduled
I20240503 02:13:47.662781 624 rebalancer_tool.cc:1074] tablet c5cd64614c53487ea1c6cc4a4ca5c379: '16440e365430405fb10366ab70e2b429' -> '9a4d92bc25374fb3960fde7d10be309d' move scheduled
I20240503 02:13:47.748044 624 rebalancer_tool.cc:1074] tablet 51f5f7297e6448d0b0e9afeedad01429: 'fbbc2be6382e47908ecb7c68c367bd26' -> '1f10f80f13c84e80b03eda7085a3b012' move scheduled
I20240503 02:13:47.801328 624 rebalancer_tool.cc:1074] tablet 48b76d7755294d37860842a4a73ab28b: 'a9094240860e43bfb129193f3bfd5be6' -> '155735c7f8b14997ac4c770aee2a9401' move scheduled
I20240503 02:13:47.871491 624 rebalancer_tool.cc:1074] tablet b76b807d2eb24f3ead50bdc0bb1b0231: 'f575ce92c0224eeb80b9fae5f6561fbe' -> '16440e365430405fb10366ab70e2b429' move scheduled
I20240503 02:13:48.160147 624 rebalancer_tool.cc:1074] tablet c97c7a843366426887115e13f35faf52: '9a4d92bc25374fb3960fde7d10be309d' -> 'fbbc2be6382e47908ecb7c68c367bd26' move scheduled
I20240503 02:13:48.460161 624 rebalancer_tool.cc:1074] tablet 9c6e204df01e450598b05c3429574700: 'f575ce92c0224eeb80b9fae5f6561fbe' -> '1f10f80f13c84e80b03eda7085a3b012' move scheduled
I20240503 02:13:48.597206 624 rebalancer_tool.cc:1074] tablet b7a2201f9605454ea12df4e3a7e5a63e: 'a9094240860e43bfb129193f3bfd5be6' -> '8a666e798fe8477d8dd1e51b385d5ead' move scheduled
I20240503 02:13:48.671283 624 rebalancer_tool.cc:1074] tablet eb70e12d4fdb4d15a93d37dfdc659a14: 'fbbc2be6382e47908ecb7c68c367bd26' -> '155735c7f8b14997ac4c770aee2a9401' move scheduled
I20240503 02:13:48.733724 624 rebalancer_tool.cc:1074] tablet 5ccde7688c4e49b5b9d89d6e048c308e: '9a4d92bc25374fb3960fde7d10be309d' -> '16440e365430405fb10366ab70e2b429' move scheduled
I20240503 02:13:57.099606 624 rebalancer_tool.cc:1136] tablet 5800022ae50043ee8b1af1e134e86e6a: '9a4d92bc25374fb3960fde7d10be309d' -> '1f10f80f13c84e80b03eda7085a3b012' move completed: OK
I20240503 02:13:57.109247 624 rebalancer_tool.cc:1136] tablet c87e7cbed0e843f5b5bb9bddef598cdb: 'f575ce92c0224eeb80b9fae5f6561fbe' -> '155735c7f8b14997ac4c770aee2a9401' move completed: OK
I20240503 02:13:57.118882 624 rebalancer_tool.cc:1136] tablet 16cf978ac35341d4adea71a17a91194c: '8a666e798fe8477d8dd1e51b385d5ead' -> '16440e365430405fb10366ab70e2b429' move completed: OK
I20240503 02:13:57.128019 624 rebalancer_tool.cc:1136] tablet 89938d4c793a44408795398078cc2ee0: '16440e365430405fb10366ab70e2b429' -> '8a666e798fe8477d8dd1e51b385d5ead' move completed: OK
I20240503 02:13:58.474545 624 rebalancer_tool.cc:1136] tablet f694fefb42d84318a6d91532105b3c90: '8a666e798fe8477d8dd1e51b385d5ead' -> 'f575ce92c0224eeb80b9fae5f6561fbe' move completed: OK
I20240503 02:13:58.531105 624 rebalancer_tool.cc:1074] tablet 5a44b34b585b47528c22fc678289b01a: '8a666e798fe8477d8dd1e51b385d5ead' -> '16440e365430405fb10366ab70e2b429' move scheduled
I20240503 02:13:58.580718 624 rebalancer_tool.cc:1074] tablet 480f7b57c2c04c1399b945a023fbe41f: '1f10f80f13c84e80b03eda7085a3b012' -> '9a4d92bc25374fb3960fde7d10be309d' move scheduled
I20240503 02:13:58.639261 624 rebalancer_tool.cc:1074] tablet fb8fcfa3f3ab46b182b692a0c4fc61ed: 'f575ce92c0224eeb80b9fae5f6561fbe' -> '8a666e798fe8477d8dd1e51b385d5ead' move scheduled
I20240503 02:13:58.718745 624 rebalancer_tool.cc:1074] tablet 28f3f05f0a344cd683cb9e010273f908: 'a9094240860e43bfb129193f3bfd5be6' -> '16440e365430405fb10366ab70e2b429' move scheduled
I20240503 02:13:58.780668 624 rebalancer_tool.cc:1074] tablet 75497fcddb62421395bd0fa7c88a406c: 'f575ce92c0224eeb80b9fae5f6561fbe' -> '155735c7f8b14997ac4c770aee2a9401' move scheduled
W20240503 02:13:59.228286 9226 meta_cache.cc:1097] Time spent processing table locations response: real 0.085s user 0.002s sys 0.000s
W20240503 02:13:59.228762 9226 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.22.108.62:40071, user_credentials={real_user=slave}} blocked reactor thread for 85236.2us
W20240503 02:13:59.229007 9261 meta_cache.cc:1261] Time spent looking up entry by key: real 0.084s user 0.005s sys 0.000s
W20240503 02:13:59.228336 9266 meta_cache.cc:1261] Time spent looking up entry by key: real 0.082s user 0.005s sys 0.000s
I20240503 02:14:03.234627 624 rebalancer_tool.cc:1136] tablet b7a2201f9605454ea12df4e3a7e5a63e: 'a9094240860e43bfb129193f3bfd5be6' -> '8a666e798fe8477d8dd1e51b385d5ead' move completed: OK
I20240503 02:14:03.246778 624 rebalancer_tool.cc:1136] tablet 9c6e204df01e450598b05c3429574700: 'f575ce92c0224eeb80b9fae5f6561fbe' -> '1f10f80f13c84e80b03eda7085a3b012' move completed: OK
I20240503 02:14:03.257236 624 rebalancer_tool.cc:1136] tablet c97c7a843366426887115e13f35faf52: '9a4d92bc25374fb3960fde7d10be309d' -> 'fbbc2be6382e47908ecb7c68c367bd26' move completed: OK
I20240503 02:14:03.266083 624 rebalancer_tool.cc:1136] tablet 48b76d7755294d37860842a4a73ab28b: 'a9094240860e43bfb129193f3bfd5be6' -> '155735c7f8b14997ac4c770aee2a9401' move completed: OK
I20240503 02:14:03.275628 624 rebalancer_tool.cc:1136] tablet c5cd64614c53487ea1c6cc4a4ca5c379: '16440e365430405fb10366ab70e2b429' -> '9a4d92bc25374fb3960fde7d10be309d' move completed: OK
I20240503 02:14:03.292084 624 rebalancer_tool.cc:1136] tablet ac4d497de05c49edbd6e0edac73c058d: 'fbbc2be6382e47908ecb7c68c367bd26' -> '155735c7f8b14997ac4c770aee2a9401' move completed: OK
I20240503 02:14:03.907316 624 rebalancer_tool.cc:1136] tablet b76b807d2eb24f3ead50bdc0bb1b0231: 'f575ce92c0224eeb80b9fae5f6561fbe' -> '16440e365430405fb10366ab70e2b429' move completed: OK
I20240503 02:14:03.942348 624 rebalancer_tool.cc:1136] tablet 4cc7861a21f94dabb01133bac9ba0b46: '9a4d92bc25374fb3960fde7d10be309d' -> '1f10f80f13c84e80b03eda7085a3b012' move completed: OK
I20240503 02:14:03.958290 624 rebalancer_tool.cc:1136] tablet c2d33db64ee8482aad840af219d60b8b: 'fbbc2be6382e47908ecb7c68c367bd26' -> 'a9094240860e43bfb129193f3bfd5be6' move completed: OK
I20240503 02:14:04.606671 624 rebalancer_tool.cc:1136] tablet 46d608d84863424e8c65feac7c2e835f: 'a9094240860e43bfb129193f3bfd5be6' -> 'f575ce92c0224eeb80b9fae5f6561fbe' move completed: OK
I20240503 02:14:04.620254 624 rebalancer_tool.cc:1136] tablet eb70e12d4fdb4d15a93d37dfdc659a14: 'fbbc2be6382e47908ecb7c68c367bd26' -> '155735c7f8b14997ac4c770aee2a9401' move completed: OK
I20240503 02:14:04.632983 624 rebalancer_tool.cc:1136] tablet 51f5f7297e6448d0b0e9afeedad01429: 'fbbc2be6382e47908ecb7c68c367bd26' -> '1f10f80f13c84e80b03eda7085a3b012' move completed: OK
I20240503 02:14:04.685304 624 rebalancer_tool.cc:1074] tablet 3c437d9d466348f2863c407666c348aa: 'fbbc2be6382e47908ecb7c68c367bd26' -> '1f10f80f13c84e80b03eda7085a3b012' move scheduled
I20240503 02:14:04.744815 624 rebalancer_tool.cc:1074] tablet 3f00949b4ca84a49b0f6e62bd7373586: 'a9094240860e43bfb129193f3bfd5be6' -> '9a4d92bc25374fb3960fde7d10be309d' move scheduled
I20240503 02:14:04.808013 624 rebalancer_tool.cc:1074] tablet 5a5d5658fe5e4104a15b404cb2022b7c: 'f575ce92c0224eeb80b9fae5f6561fbe' -> 'fbbc2be6382e47908ecb7c68c367bd26' move scheduled
I20240503 02:14:04.879063 624 rebalancer_tool.cc:1074] tablet e80b43971dc246d9bfd5e1c603f8415c: '1f10f80f13c84e80b03eda7085a3b012' -> '9a4d92bc25374fb3960fde7d10be309d' move scheduled
I20240503 02:14:04.985380 624 rebalancer_tool.cc:1074] tablet 6635aacafed54994965ad48b2df4f163: 'a9094240860e43bfb129193f3bfd5be6' -> '8a666e798fe8477d8dd1e51b385d5ead' move scheduled
I20240503 02:14:05.088378 624 rebalancer_tool.cc:1074] tablet eec091fa0ba947d68577d562e538ab56: 'fbbc2be6382e47908ecb7c68c367bd26' -> '8a666e798fe8477d8dd1e51b385d5ead' move scheduled
I20240503 02:14:05.152666 624 rebalancer_tool.cc:1074] tablet c19017f829a74cc2b6dc41a2a25a3831: '16440e365430405fb10366ab70e2b429' -> 'f575ce92c0224eeb80b9fae5f6561fbe' move scheduled
I20240503 02:14:05.332031 624 rebalancer_tool.cc:1074] tablet 0007679ffe8047d79dd8ac778fb6f1ce: '1f10f80f13c84e80b03eda7085a3b012' -> '9a4d92bc25374fb3960fde7d10be309d' move scheduled
I20240503 02:14:05.448463 624 rebalancer_tool.cc:1074] tablet 2bf8c34ee16d4e0fbc6817d63b2ba52e: 'a9094240860e43bfb129193f3bfd5be6' -> 'f575ce92c0224eeb80b9fae5f6561fbe' move scheduled
I20240503 02:14:12.039005 624 rebalancer_tool.cc:1136] tablet fb8fcfa3f3ab46b182b692a0c4fc61ed: 'f575ce92c0224eeb80b9fae5f6561fbe' -> '8a666e798fe8477d8dd1e51b385d5ead' move completed: OK
I20240503 02:14:12.065683 624 rebalancer_tool.cc:1136] tablet 480f7b57c2c04c1399b945a023fbe41f: '1f10f80f13c84e80b03eda7085a3b012' -> '9a4d92bc25374fb3960fde7d10be309d' move completed: OK
I20240503 02:14:13.441901 624 rebalancer_tool.cc:1136] tablet 5ccde7688c4e49b5b9d89d6e048c308e: '9a4d92bc25374fb3960fde7d10be309d' -> '16440e365430405fb10366ab70e2b429' move completed: OK
I20240503 02:14:13.473321 624 rebalancer_tool.cc:1136] tablet 75497fcddb62421395bd0fa7c88a406c: 'f575ce92c0224eeb80b9fae5f6561fbe' -> '155735c7f8b14997ac4c770aee2a9401' move completed: OK
I20240503 02:14:13.488394 624 rebalancer_tool.cc:1136] tablet db0756f0f331402aaef0583e1f9e5318: '155735c7f8b14997ac4c770aee2a9401' -> '8a666e798fe8477d8dd1e51b385d5ead' move completed: OK
I20240503 02:14:13.540302 624 rebalancer_tool.cc:1074] tablet 9b3b9d83e4c742bfa62624676cbb6a11: 'fbbc2be6382e47908ecb7c68c367bd26' -> '8a666e798fe8477d8dd1e51b385d5ead' move scheduled
I20240503 02:14:13.600196 624 rebalancer_tool.cc:1074] tablet ba97049157a646c780fa203d34cd4c82: '1f10f80f13c84e80b03eda7085a3b012' -> '9a4d92bc25374fb3960fde7d10be309d' move scheduled
I20240503 02:14:13.709784 624 rebalancer_tool.cc:1074] tablet f82e505ac13a41019e422b9dc1e4fef9: '16440e365430405fb10366ab70e2b429' -> 'f575ce92c0224eeb80b9fae5f6561fbe' move scheduled
I20240503 02:14:13.762167 624 rebalancer_tool.cc:1074] tablet 683d4041910f4355910e188c0b7df6c7: 'a9094240860e43bfb129193f3bfd5be6' -> '8a666e798fe8477d8dd1e51b385d5ead' move scheduled
I20240503 02:14:13.810379 624 rebalancer_tool.cc:1074] tablet 3351f8900c7446179da1fbe65f835e0a: '9a4d92bc25374fb3960fde7d10be309d' -> '16440e365430405fb10366ab70e2b429' move scheduled
I20240503 02:14:15.807910 624 rebalancer_tool.cc:1136] tablet c19017f829a74cc2b6dc41a2a25a3831: '16440e365430405fb10366ab70e2b429' -> 'f575ce92c0224eeb80b9fae5f6561fbe' move completed: OK
I20240503 02:14:15.819633 624 rebalancer_tool.cc:1136] tablet eec091fa0ba947d68577d562e538ab56: 'fbbc2be6382e47908ecb7c68c367bd26' -> '8a666e798fe8477d8dd1e51b385d5ead' move completed: OK
I20240503 02:14:15.831821 624 rebalancer_tool.cc:1136] tablet 6635aacafed54994965ad48b2df4f163: 'a9094240860e43bfb129193f3bfd5be6' -> '8a666e798fe8477d8dd1e51b385d5ead' move completed: OK
I20240503 02:14:16.536581 624 rebalancer_tool.cc:1136] tablet e80b43971dc246d9bfd5e1c603f8415c: '1f10f80f13c84e80b03eda7085a3b012' -> '9a4d92bc25374fb3960fde7d10be309d' move completed: OK
I20240503 02:14:16.549383 624 rebalancer_tool.cc:1136] tablet 5a5d5658fe5e4104a15b404cb2022b7c: 'f575ce92c0224eeb80b9fae5f6561fbe' -> 'fbbc2be6382e47908ecb7c68c367bd26' move completed: OK
I20240503 02:14:16.574236 624 rebalancer_tool.cc:1136] tablet 3f00949b4ca84a49b0f6e62bd7373586: 'a9094240860e43bfb129193f3bfd5be6' -> '9a4d92bc25374fb3960fde7d10be309d' move completed: OK
I20240503 02:14:16.583578 624 rebalancer_tool.cc:1136] tablet 3c437d9d466348f2863c407666c348aa: 'fbbc2be6382e47908ecb7c68c367bd26' -> '1f10f80f13c84e80b03eda7085a3b012' move completed: OK
I20240503 02:14:16.597579 624 rebalancer_tool.cc:1136] tablet 28f3f05f0a344cd683cb9e010273f908: 'a9094240860e43bfb129193f3bfd5be6' -> '16440e365430405fb10366ab70e2b429' move completed: OK
I20240503 02:14:17.238262 624 rebalancer_tool.cc:1136] tablet 5a44b34b585b47528c22fc678289b01a: '8a666e798fe8477d8dd1e51b385d5ead' -> '16440e365430405fb10366ab70e2b429' move completed: OK
I20240503 02:14:17.894804 624 rebalancer_tool.cc:1136] tablet 0007679ffe8047d79dd8ac778fb6f1ce: '1f10f80f13c84e80b03eda7085a3b012' -> '9a4d92bc25374fb3960fde7d10be309d' move completed: OK
I20240503 02:14:17.963191 624 rebalancer_tool.cc:1074] tablet eb0f4fe6b8684f329725ac2f7d5428f2: '8a666e798fe8477d8dd1e51b385d5ead' -> 'fbbc2be6382e47908ecb7c68c367bd26' move scheduled
I20240503 02:14:18.040434 624 rebalancer_tool.cc:1074] tablet e8a76caac93d441b92e65380bf6dc03e: 'f575ce92c0224eeb80b9fae5f6561fbe' -> 'a9094240860e43bfb129193f3bfd5be6' move scheduled
I20240503 02:14:18.113529 624 rebalancer_tool.cc:1074] tablet 08e4d119d90f4146ba53a1b18efe2386: '9a4d92bc25374fb3960fde7d10be309d' -> '16440e365430405fb10366ab70e2b429' move scheduled
I20240503 02:14:18.221767 624 rebalancer_tool.cc:1074] tablet 9be5395489a3488da62f4d5b66facf3d: '8a666e798fe8477d8dd1e51b385d5ead' -> 'a9094240860e43bfb129193f3bfd5be6' move scheduled
I20240503 02:14:18.294726 624 rebalancer_tool.cc:1074] tablet fa288463ae154e28afd9c71b4b55ef3a: 'a9094240860e43bfb129193f3bfd5be6' -> 'f575ce92c0224eeb80b9fae5f6561fbe' move scheduled
I20240503 02:14:18.354086 624 rebalancer_tool.cc:1074] tablet 62d5e9ea7af8435bb34028df2b3f1457: '9a4d92bc25374fb3960fde7d10be309d' -> '16440e365430405fb10366ab70e2b429' move scheduled
I20240503 02:14:18.437969 624 rebalancer_tool.cc:1074] tablet 88f696d03330448d879ed7b60364c28c: '9a4d92bc25374fb3960fde7d10be309d' -> '16440e365430405fb10366ab70e2b429' move scheduled
I20240503 02:14:23.818499 624 rebalancer_tool.cc:1136] tablet 9b3b9d83e4c742bfa62624676cbb6a11: 'fbbc2be6382e47908ecb7c68c367bd26' -> '8a666e798fe8477d8dd1e51b385d5ead' move completed: OK
I20240503 02:14:23.831382 624 rebalancer_tool.cc:1136] tablet 2bf8c34ee16d4e0fbc6817d63b2ba52e: 'a9094240860e43bfb129193f3bfd5be6' -> 'f575ce92c0224eeb80b9fae5f6561fbe' move completed: OK
I20240503 02:14:23.857069 624 rebalancer_tool.cc:1136] tablet 683d4041910f4355910e188c0b7df6c7: 'a9094240860e43bfb129193f3bfd5be6' -> '8a666e798fe8477d8dd1e51b385d5ead' move completed: OK
I20240503 02:14:23.873651 624 rebalancer_tool.cc:1136] tablet 3351f8900c7446179da1fbe65f835e0a: '9a4d92bc25374fb3960fde7d10be309d' -> '16440e365430405fb10366ab70e2b429' move completed: OK
I20240503 02:14:23.883870 624 rebalancer_tool.cc:1136] tablet f82e505ac13a41019e422b9dc1e4fef9: '16440e365430405fb10366ab70e2b429' -> 'f575ce92c0224eeb80b9fae5f6561fbe' move completed: OK
I20240503 02:14:23.960374 624 rebalancer_tool.cc:1074] tablet 4055b2f8d72041acaa81e6e98db9042c: '8a666e798fe8477d8dd1e51b385d5ead' -> 'a9094240860e43bfb129193f3bfd5be6' move scheduled
I20240503 02:14:24.035339 624 rebalancer_tool.cc:1074] tablet 331905034afd48c19eb2bc2e2ec40280: '9a4d92bc25374fb3960fde7d10be309d' -> '16440e365430405fb10366ab70e2b429' move scheduled
I20240503 02:14:24.091346 624 rebalancer_tool.cc:1074] tablet a52bee1fdc9d48b1a3463ecd62c8b44e: '8a666e798fe8477d8dd1e51b385d5ead' -> 'a9094240860e43bfb129193f3bfd5be6' move scheduled
I20240503 02:14:26.067377 624 rebalancer_tool.cc:1136] tablet 62d5e9ea7af8435bb34028df2b3f1457: '9a4d92bc25374fb3960fde7d10be309d' -> '16440e365430405fb10366ab70e2b429' move completed: OK
I20240503 02:14:26.078428 624 rebalancer_tool.cc:1136] tablet fa288463ae154e28afd9c71b4b55ef3a: 'a9094240860e43bfb129193f3bfd5be6' -> 'f575ce92c0224eeb80b9fae5f6561fbe' move completed: OK
I20240503 02:14:27.206918 624 rebalancer_tool.cc:1136] tablet 08e4d119d90f4146ba53a1b18efe2386: '9a4d92bc25374fb3960fde7d10be309d' -> '16440e365430405fb10366ab70e2b429' move completed: OK
I20240503 02:14:27.257004 624 rebalancer_tool.cc:1136] tablet e8a76caac93d441b92e65380bf6dc03e: 'f575ce92c0224eeb80b9fae5f6561fbe' -> 'a9094240860e43bfb129193f3bfd5be6' move completed: OK
I20240503 02:14:27.365821 624 rebalancer_tool.cc:1136] tablet eb0f4fe6b8684f329725ac2f7d5428f2: '8a666e798fe8477d8dd1e51b385d5ead' -> 'fbbc2be6382e47908ecb7c68c367bd26' move completed: OK
I20240503 02:14:27.491359 624 rebalancer_tool.cc:1136] tablet ba97049157a646c780fa203d34cd4c82: '1f10f80f13c84e80b03eda7085a3b012' -> '9a4d92bc25374fb3960fde7d10be309d' move completed: OK
I20240503 02:14:27.600690 624 rebalancer_tool.cc:1136] tablet 9be5395489a3488da62f4d5b66facf3d: '8a666e798fe8477d8dd1e51b385d5ead' -> 'a9094240860e43bfb129193f3bfd5be6' move completed: OK
I20240503 02:14:27.877175 624 rebalancer_tool.cc:1074] tablet 2baf70e0e1894cea9fdeb92279bc0cff: '9a4d92bc25374fb3960fde7d10be309d' -> 'a9094240860e43bfb129193f3bfd5be6' move scheduled
I20240503 02:14:28.257277 624 rebalancer_tool.cc:1074] tablet a6505f95a1c547748fb5cb7a4918b129: '8a666e798fe8477d8dd1e51b385d5ead' -> 'a9094240860e43bfb129193f3bfd5be6' move scheduled
I20240503 02:14:28.488502 624 rebalancer_tool.cc:1074] tablet 737a39a4f65b4e9ab2ce65c0d7f62dcb: '9a4d92bc25374fb3960fde7d10be309d' -> 'a9094240860e43bfb129193f3bfd5be6' move scheduled
I20240503 02:14:31.259850 624 rebalancer_tool.cc:1136] tablet 4055b2f8d72041acaa81e6e98db9042c: '8a666e798fe8477d8dd1e51b385d5ead' -> 'a9094240860e43bfb129193f3bfd5be6' move completed: OK
I20240503 02:14:31.272781 624 rebalancer_tool.cc:1136] tablet 88f696d03330448d879ed7b60364c28c: '9a4d92bc25374fb3960fde7d10be309d' -> '16440e365430405fb10366ab70e2b429' move completed: OK
I20240503 02:14:31.288252 624 rebalancer_tool.cc:1136] tablet a52bee1fdc9d48b1a3463ecd62c8b44e: '8a666e798fe8477d8dd1e51b385d5ead' -> 'a9094240860e43bfb129193f3bfd5be6' move completed: OK
I20240503 02:14:31.351972 624 rebalancer_tool.cc:1074] tablet 4563386631774e74a4b12fc71714c1a0: '9a4d92bc25374fb3960fde7d10be309d' -> 'a9094240860e43bfb129193f3bfd5be6' move scheduled
I20240503 02:14:32.789063 624 rebalancer_tool.cc:1136] tablet 2baf70e0e1894cea9fdeb92279bc0cff: '9a4d92bc25374fb3960fde7d10be309d' -> 'a9094240860e43bfb129193f3bfd5be6' move completed: OK
I20240503 02:14:32.798792 624 rebalancer_tool.cc:1136] tablet 737a39a4f65b4e9ab2ce65c0d7f62dcb: '9a4d92bc25374fb3960fde7d10be309d' -> 'a9094240860e43bfb129193f3bfd5be6' move completed: OK
I20240503 02:14:32.809418 624 rebalancer_tool.cc:1136] tablet 331905034afd48c19eb2bc2e2ec40280: '9a4d92bc25374fb3960fde7d10be309d' -> '16440e365430405fb10366ab70e2b429' move completed: OK
I20240503 02:14:33.519758 624 rebalancer_tool.cc:1136] tablet a6505f95a1c547748fb5cb7a4918b129: '8a666e798fe8477d8dd1e51b385d5ead' -> 'a9094240860e43bfb129193f3bfd5be6' move completed: OK
I20240503 02:14:33.530043 624 rebalancer_tool.cc:1136] tablet 4563386631774e74a4b12fc71714c1a0: '9a4d92bc25374fb3960fde7d10be309d' -> 'a9094240860e43bfb129193f3bfd5be6' move completed: OK
I20240503 02:14:34.363679 624 rebalancer_tool.cc:1074] tablet 52a541b1d3db4228bc5a27af521f0b93: '16440e365430405fb10366ab70e2b429' -> '1f10f80f13c84e80b03eda7085a3b012' move scheduled
I20240503 02:14:34.598063 624 rebalancer_tool.cc:1074] tablet 5faccb32e55d4cb689ff491fd20884e6: '155735c7f8b14997ac4c770aee2a9401' -> 'f575ce92c0224eeb80b9fae5f6561fbe' move scheduled
I20240503 02:14:34.702955 624 rebalancer_tool.cc:1074] tablet 32a2dbf80daf40d6b21442a39f62f14c: 'a9094240860e43bfb129193f3bfd5be6' -> 'fbbc2be6382e47908ecb7c68c367bd26' move scheduled
I20240503 02:14:34.755399 624 rebalancer_tool.cc:1074] tablet fa288463ae154e28afd9c71b4b55ef3a: '16440e365430405fb10366ab70e2b429' -> '8a666e798fe8477d8dd1e51b385d5ead' move scheduled
I20240503 02:14:34.819080 624 rebalancer_tool.cc:1074] tablet 04e47da8cf7445dcacaf1f7c8950bd34: '1f10f80f13c84e80b03eda7085a3b012' -> '9a4d92bc25374fb3960fde7d10be309d' move scheduled
I20240503 02:14:34.921855 624 rebalancer_tool.cc:1074] tablet 5cf58a111e3048449303ec45ac091b42: 'f575ce92c0224eeb80b9fae5f6561fbe' -> '155735c7f8b14997ac4c770aee2a9401' move scheduled
I20240503 02:14:34.987247 624 rebalancer_tool.cc:1074] tablet 0007679ffe8047d79dd8ac778fb6f1ce: 'a9094240860e43bfb129193f3bfd5be6' -> '8a666e798fe8477d8dd1e51b385d5ead' move scheduled
I20240503 02:14:35.051746 624 rebalancer_tool.cc:1074] tablet 8f5e6ff35b994d46a1e1fe216eb356ce: 'fbbc2be6382e47908ecb7c68c367bd26' -> '9a4d92bc25374fb3960fde7d10be309d' move scheduled
I20240503 02:14:35.151333 624 rebalancer_tool.cc:1074] tablet 8929e26a53274cecbb808dbd8b203a64: '16440e365430405fb10366ab70e2b429' -> 'a9094240860e43bfb129193f3bfd5be6' move scheduled
I20240503 02:14:35.299086 624 rebalancer_tool.cc:1074] tablet 58e1aaecf47944569296fc1e0780efb5: '1f10f80f13c84e80b03eda7085a3b012' -> 'f575ce92c0224eeb80b9fae5f6561fbe' move scheduled
I20240503 02:14:35.434545 624 rebalancer_tool.cc:1074] tablet 88b57085de3144cea3f64922ae5ccf55: '155735c7f8b14997ac4c770aee2a9401' -> 'fbbc2be6382e47908ecb7c68c367bd26' move scheduled
I20240503 02:14:35.698343 624 rebalancer_tool.cc:1074] tablet e80b43971dc246d9bfd5e1c603f8415c: '16440e365430405fb10366ab70e2b429' -> '8a666e798fe8477d8dd1e51b385d5ead' move scheduled
I20240503 02:14:35.782392 624 rebalancer_tool.cc:1074] tablet b90b887127f44c0d920225be10077c64: 'a9094240860e43bfb129193f3bfd5be6' -> '9a4d92bc25374fb3960fde7d10be309d' move scheduled
I20240503 02:14:35.911301 624 rebalancer_tool.cc:1074] tablet b7b284cffcb04ce883a24f6a8796748f: 'fbbc2be6382e47908ecb7c68c367bd26' -> '1f10f80f13c84e80b03eda7085a3b012' move scheduled
I20240503 02:14:36.027602 624 rebalancer_tool.cc:1074] tablet 9cd1418eb38d420c9c41d660476eafb0: 'f575ce92c0224eeb80b9fae5f6561fbe' -> '155735c7f8b14997ac4c770aee2a9401' move scheduled
I20240503 02:14:36.100998 624 rebalancer_tool.cc:1074] tablet 08e4d119d90f4146ba53a1b18efe2386: '8a666e798fe8477d8dd1e51b385d5ead' -> 'a9094240860e43bfb129193f3bfd5be6' move scheduled
I20240503 02:14:36.163378 624 rebalancer_tool.cc:1074] tablet 35a5a7f4fe184b9fadc94c98b61deaf0: '16440e365430405fb10366ab70e2b429' -> '9a4d92bc25374fb3960fde7d10be309d' move scheduled
I20240503 02:14:36.337644 624 rebalancer_tool.cc:1074] tablet b8b119c0d8044f2ab19b2100950270d6: 'f575ce92c0224eeb80b9fae5f6561fbe' -> 'fbbc2be6382e47908ecb7c68c367bd26' move scheduled
I20240503 02:14:36.393816 624 rebalancer_tool.cc:1074] tablet 6635aacafed54994965ad48b2df4f163: '1f10f80f13c84e80b03eda7085a3b012' -> '9a4d92bc25374fb3960fde7d10be309d' move scheduled
W20240503 02:14:37.552536 14520 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:14:37.384228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.108.5:35595 (local address 127.0.0.1:58110)
0503 02:14:37.385091 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0503 02:14:37.385108 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0503 02:14:37.385205 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:14:37.551079 (+165874us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":711}
I20240503 02:14:50.932024 624 rebalancer_tool.cc:1136] tablet 08e4d119d90f4146ba53a1b18efe2386: '8a666e798fe8477d8dd1e51b385d5ead' -> 'a9094240860e43bfb129193f3bfd5be6' move completed: OK
I20240503 02:14:53.316490 624 rebalancer_tool.cc:1136] tablet 88b57085de3144cea3f64922ae5ccf55: '155735c7f8b14997ac4c770aee2a9401' -> 'fbbc2be6382e47908ecb7c68c367bd26' move completed: OK
I20240503 02:14:53.331772 624 rebalancer_tool.cc:1136] tablet 8929e26a53274cecbb808dbd8b203a64: '16440e365430405fb10366ab70e2b429' -> 'a9094240860e43bfb129193f3bfd5be6' move completed: OK
I20240503 02:14:54.307068 624 rebalancer_tool.cc:1136] tablet 5cf58a111e3048449303ec45ac091b42: 'f575ce92c0224eeb80b9fae5f6561fbe' -> '155735c7f8b14997ac4c770aee2a9401' move completed: OK
I20240503 02:14:55.477254 624 rebalancer_tool.cc:1136] tablet fa288463ae154e28afd9c71b4b55ef3a: '16440e365430405fb10366ab70e2b429' -> '8a666e798fe8477d8dd1e51b385d5ead' move completed: OK
I20240503 02:14:55.505211 624 rebalancer_tool.cc:1136] tablet 32a2dbf80daf40d6b21442a39f62f14c: 'a9094240860e43bfb129193f3bfd5be6' -> 'fbbc2be6382e47908ecb7c68c367bd26' move completed: OK
I20240503 02:14:55.981741 624 rebalancer_tool.cc:1136] tablet 5faccb32e55d4cb689ff491fd20884e6: '155735c7f8b14997ac4c770aee2a9401' -> 'f575ce92c0224eeb80b9fae5f6561fbe' move completed: OK
I20240503 02:14:57.622102 624 rebalancer_tool.cc:1136] tablet 52a541b1d3db4228bc5a27af521f0b93: '16440e365430405fb10366ab70e2b429' -> '1f10f80f13c84e80b03eda7085a3b012' move completed: OK
I20240503 02:14:58.715634 624 rebalancer_tool.cc:1074] tablet c5cd64614c53487ea1c6cc4a4ca5c379: '155735c7f8b14997ac4c770aee2a9401' -> 'fbbc2be6382e47908ecb7c68c367bd26' move scheduled
I20240503 02:14:58.806650 624 rebalancer_tool.cc:1074] tablet 2baf70e0e1894cea9fdeb92279bc0cff: '8a666e798fe8477d8dd1e51b385d5ead' -> 'f575ce92c0224eeb80b9fae5f6561fbe' move scheduled
I20240503 02:14:59.806661 624 rebalancer_tool.cc:1074] tablet 9b3b9d83e4c742bfa62624676cbb6a11: 'a9094240860e43bfb129193f3bfd5be6' -> '1f10f80f13c84e80b03eda7085a3b012' move scheduled
I20240503 02:14:59.894589 624 rebalancer_tool.cc:1074] tablet ebcd2c6e1722442fbe603cfcccb56ab0: 'fbbc2be6382e47908ecb7c68c367bd26' -> '155735c7f8b14997ac4c770aee2a9401' move scheduled
I20240503 02:15:59.931198 624 rebalancer_tool.cc:1084] tablet 6bf84757d1fb44689da830a82e857787: 'a9094240860e43bfb129193f3bfd5be6' -> 'f575ce92c0224eeb80b9fae5f6561fbe' move ignored: Timed out: ChangeConfig RPC to 127.22.108.4:33935 timed out after 60.000s (SENT)
I20240503 02:15:59.931347 624 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): unacceptable health status UNAVAILABLE
I20240503 02:16:01.178347 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 29793
I20240503 02:16:01.187335 29914 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:16:01.188494 30199 consensus_peers.cc:485] T fa288463ae154e28afd9c71b4b55ef3a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:16:01.237236 32709 consensus_peers.cc:485] T e530c3ca50384a7c863001eb6d3fc63a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:16:01.299852 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:16:01.332047 30070 consensus_peers.cc:485] T 9b3b9d83e4c742bfa62624676cbb6a11 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:16:01.373051 370 consensus_peers.cc:485] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:16:01.407158 29940 consensus_peers.cc:485] T 04e47da8cf7445dcacaf1f7c8950bd34 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:16:01.464898 29939 consensus_peers.cc:485] T c42a89fd9daf427bb5a60b4ee76a9287 P 155735c7f8b14997ac4c770aee2a9401 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:16:01.467396 371 consensus_peers.cc:485] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:16:01.527496 30069 consensus_peers.cc:485] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:16:01.536433 30200 consensus_peers.cc:485] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:16:01.612188 29809 consensus_peers.cc:485] T 28f3f05f0a344cd683cb9e010273f908 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:16:01.613893 30199 consensus_peers.cc:485] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:16:01.618113 29809 consensus_peers.cc:485] T 3f00949b4ca84a49b0f6e62bd7373586 P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:16:01.643422 29809 consensus_peers.cc:485] T 75497fcddb62421395bd0fa7c88a406c P 16440e365430405fb10366ab70e2b429 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:16:01.652832 29940 consensus_peers.cc:462] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:16:01.655416 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 29793
W20240503 02:16:01.677105 370 consensus_peers.cc:485] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:16:01.685333 29940 consensus_peers.cc:462] T 6635aacafed54994965ad48b2df4f163 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
W20240503 02:16:01.725597 30201 connection.cc:535] server connection from 127.22.108.1:38817 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:16:01.725653 373 connection.cc:535] client connection to 127.22.108.1:36719 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:16:01.725980 29942 connection.cc:535] client connection to 127.22.108.1:36719 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:16:01.727070 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 29924
W20240503 02:16:01.727514 30072 connection.cc:535] client connection to 127.22.108.1:36719 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:16:01.727574 29942 consensus_peers.cc:485] T e61579e74a1a4a9dbd9a09f9b34d9c8d P 155735c7f8b14997ac4c770aee2a9401 -> Peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Couldn't send request to peer 16440e365430405fb10366ab70e2b429. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:01.727660 29942 consensus_peers.cc:485] T 35a5a7f4fe184b9fadc94c98b61deaf0 P 155735c7f8b14997ac4c770aee2a9401 -> Peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Couldn't send request to peer 16440e365430405fb10366ab70e2b429. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:01.727720 29942 consensus_peers.cc:485] T b7a2201f9605454ea12df4e3a7e5a63e P 155735c7f8b14997ac4c770aee2a9401 -> Peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Couldn't send request to peer 16440e365430405fb10366ab70e2b429. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:01.727705 30202 consensus_peers.cc:485] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Couldn't send request to peer 16440e365430405fb10366ab70e2b429. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:01.727797 29942 consensus_peers.cc:485] T 63b416f4e004490b8e38a96fd065dfea P 155735c7f8b14997ac4c770aee2a9401 -> Peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Couldn't send request to peer 16440e365430405fb10366ab70e2b429. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:01.728205 373 consensus_peers.cc:485] T 5ccde7688c4e49b5b9d89d6e048c308e P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Couldn't send request to peer 16440e365430405fb10366ab70e2b429. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:01.728314 373 consensus_peers.cc:485] T eb0f4fe6b8684f329725ac2f7d5428f2 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Couldn't send request to peer 16440e365430405fb10366ab70e2b429. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:01.728392 373 consensus_peers.cc:485] T 3351f8900c7446179da1fbe65f835e0a P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Couldn't send request to peer 16440e365430405fb10366ab70e2b429. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:01.728452 373 consensus_peers.cc:485] T 88f696d03330448d879ed7b60364c28c P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Couldn't send request to peer 16440e365430405fb10366ab70e2b429. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:01.728521 373 consensus_peers.cc:485] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Couldn't send request to peer 16440e365430405fb10366ab70e2b429. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:01.728590 373 consensus_peers.cc:485] T ec3891bfc4014b63afef6c259de00ca7 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Couldn't send request to peer 16440e365430405fb10366ab70e2b429. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:01.728655 373 consensus_peers.cc:485] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Couldn't send request to peer 16440e365430405fb10366ab70e2b429. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:01.728767 30072 consensus_peers.cc:485] T e80b43971dc246d9bfd5e1c603f8415c P a9094240860e43bfb129193f3bfd5be6 -> Peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Couldn't send request to peer 16440e365430405fb10366ab70e2b429. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:01.728895 30072 consensus_peers.cc:485] T 5040203bd48549ff9c8f64c71d6bc598 P a9094240860e43bfb129193f3bfd5be6 -> Peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Couldn't send request to peer 16440e365430405fb10366ab70e2b429. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:01.728986 30072 consensus_peers.cc:485] T 1b6cda34c02e4d24a126171c0063a7d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Couldn't send request to peer 16440e365430405fb10366ab70e2b429. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:01.729429 29940 connection.cc:535] server connection from 127.22.108.1:41771 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:16:01.726507 30069 connection.cc:535] server connection from 127.22.108.1:55443 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:16:01.730691 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
I20240503 02:16:01.735131 30045 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:16:01.738898 30202 consensus_peers.cc:485] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Couldn't send request to peer 16440e365430405fb10366ab70e2b429. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:01.778627 30202 consensus_peers.cc:485] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Couldn't send request to peer 16440e365430405fb10366ab70e2b429. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:01.804216 371 consensus_peers.cc:485] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:16:01.847983 30070 consensus_peers.cc:485] T b8b119c0d8044f2ab19b2100950270d6 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:16:01.869608 30202 consensus_peers.cc:485] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Couldn't send request to peer 16440e365430405fb10366ab70e2b429. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:01.892891 30069 consensus_peers.cc:485] T 88b57085de3144cea3f64922ae5ccf55 P a9094240860e43bfb129193f3bfd5be6 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:16:01.908751 371 consensus_peers.cc:485] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:16:01.934791 30200 consensus_peers.cc:485] T fb8fcfa3f3ab46b182b692a0c4fc61ed P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:16:01.963265 30072 consensus_peers.cc:485] T c2d33db64ee8482aad840af219d60b8b P a9094240860e43bfb129193f3bfd5be6 -> Peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Couldn't send request to peer 16440e365430405fb10366ab70e2b429. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:01.992663 30072 consensus_peers.cc:485] T b6c9fd3cbbd5453897aa072600db569d P a9094240860e43bfb129193f3bfd5be6 -> Peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Couldn't send request to peer 16440e365430405fb10366ab70e2b429. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:02.021581 30072 consensus_peers.cc:485] T 88509fbaa62645c58b7ab1a432eb8727 P a9094240860e43bfb129193f3bfd5be6 -> Peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Couldn't send request to peer 16440e365430405fb10366ab70e2b429. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:02.027912 32711 consensus_peers.cc:485] T 331905034afd48c19eb2bc2e2ec40280 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Couldn't send request to peer 16440e365430405fb10366ab70e2b429. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:02.030069 30070 consensus_peers.cc:485] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:16:02.055050 32709 consensus_peers.cc:485] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:16:02.073876 30199 consensus_peers.cc:485] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:16:02.098397 32708 consensus_peers.cc:485] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:16:02.123256 30070 consensus_peers.cc:485] T 2a132e4747824d5989858e649bc243c2 P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 02:16:02.124065 17449 raft_consensus.cc:482] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f575ce92c0224eeb80b9fae5f6561fbe)
I20240503 02:16:02.124235 17449 raft_consensus.cc:504] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:16:02.124852 17449 leader_election.cc:288] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
W20240503 02:16:02.126382 32708 leader_election.cc:334] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:16:02.126494 32709 leader_election.cc:334] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:16:02.126613 32709 leader_election.cc:302] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead; no voters: 9a4d92bc25374fb3960fde7d10be309d, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:16:02.126834 17449 raft_consensus.cc:2726] T eec091fa0ba947d68577d562e538ab56 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:16:02.143798 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 29924
W20240503 02:16:02.158798 30070 consensus_peers.cc:485] T e8a76caac93d441b92e65380bf6dc03e P a9094240860e43bfb129193f3bfd5be6 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:16:02.170068 30202 consensus_peers.cc:485] T 5a44b34b585b47528c22fc678289b01a P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Couldn't send request to peer 16440e365430405fb10366ab70e2b429. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:02.197738 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:16:02.209375 32709 connection.cc:535] client connection to 127.22.108.2:37873 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:16:02.209455 32710 connection.cc:535] server connection from 127.22.108.2:53863 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:16:02.209441 30200 connection.cc:535] server connection from 127.22.108.2:41815 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:16:02.209623 30200 connection.cc:535] client connection to 127.22.108.2:37873 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:16:02.210011 370 connection.cc:535] server connection from 127.22.108.2:48617 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:16:02.210934 32709 consensus_peers.cc:485] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Couldn't send request to peer 155735c7f8b14997ac4c770aee2a9401. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.2:37873: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:02.210955 371 connection.cc:535] client connection to 127.22.108.2:37873 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:16:02.211062 32709 consensus_peers.cc:485] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Couldn't send request to peer 155735c7f8b14997ac4c770aee2a9401. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.2:37873: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:02.211134 32709 consensus_peers.cc:485] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Couldn't send request to peer 155735c7f8b14997ac4c770aee2a9401. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.2:37873: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:02.211199 32709 consensus_peers.cc:485] T 331905034afd48c19eb2bc2e2ec40280 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Couldn't send request to peer 155735c7f8b14997ac4c770aee2a9401. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.2:37873: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:02.211256 32709 consensus_peers.cc:485] T 00aedc4902a745d6a859cbbf99529a7f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Couldn't send request to peer 155735c7f8b14997ac4c770aee2a9401. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.2:37873: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:16:02.211253 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 30054
W20240503 02:16:02.211309 32709 consensus_peers.cc:485] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Couldn't send request to peer 155735c7f8b14997ac4c770aee2a9401. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.2:37873: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:02.211369 32709 consensus_peers.cc:485] T ebcd2c6e1722442fbe603cfcccb56ab0 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Couldn't send request to peer 155735c7f8b14997ac4c770aee2a9401. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.2:37873: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:02.211427 32709 consensus_peers.cc:485] T c87e7cbed0e843f5b5bb9bddef598cdb P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Couldn't send request to peer 155735c7f8b14997ac4c770aee2a9401. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.2:37873: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:02.211488 32709 consensus_peers.cc:485] T 368d683f40cd4d0994e7bbe90e045d16 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Couldn't send request to peer 155735c7f8b14997ac4c770aee2a9401. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.2:37873: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:02.211555 32709 consensus_peers.cc:485] T 480f7b57c2c04c1399b945a023fbe41f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Couldn't send request to peer 155735c7f8b14997ac4c770aee2a9401. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.2:37873: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:02.212519 371 consensus_peers.cc:485] T 4cc7861a21f94dabb01133bac9ba0b46 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Couldn't send request to peer 155735c7f8b14997ac4c770aee2a9401. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.2:37873: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:02.212651 371 consensus_peers.cc:485] T 459c1f3a63b1442191982070ea04cc87 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Couldn't send request to peer 155735c7f8b14997ac4c770aee2a9401. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.2:37873: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:02.212744 371 consensus_peers.cc:485] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Couldn't send request to peer 155735c7f8b14997ac4c770aee2a9401. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.2:37873: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:02.212836 371 consensus_peers.cc:485] T 83e8dad806564b128d5f0fe559d8ffe1 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Couldn't send request to peer 155735c7f8b14997ac4c770aee2a9401. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.2:37873: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:02.212877 30200 consensus_peers.cc:485] T 89938d4c793a44408795398078cc2ee0 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Couldn't send request to peer 155735c7f8b14997ac4c770aee2a9401. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.2:37873: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:02.212976 30200 consensus_peers.cc:485] T e61f7581952e4d2ab0a473d2f965ba15 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Couldn't send request to peer 155735c7f8b14997ac4c770aee2a9401. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.2:37873: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:02.213083 30200 consensus_peers.cc:485] T 683d4041910f4355910e188c0b7df6c7 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Couldn't send request to peer 155735c7f8b14997ac4c770aee2a9401. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.2:37873: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:02.214814 30070 consensus_peers.cc:485] T 7076453cbe5944eb9b075127e1e0495d P a9094240860e43bfb129193f3bfd5be6 -> Peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Couldn't send request to peer 155735c7f8b14997ac4c770aee2a9401. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.2:37873: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:02.214943 30070 consensus_peers.cc:485] T c5cd64614c53487ea1c6cc4a4ca5c379 P a9094240860e43bfb129193f3bfd5be6 -> Peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Couldn't send request to peer 155735c7f8b14997ac4c770aee2a9401. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.2:37873: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:02.215049 30070 consensus_peers.cc:485] T 5a5d5658fe5e4104a15b404cb2022b7c P a9094240860e43bfb129193f3bfd5be6 -> Peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Couldn't send request to peer 155735c7f8b14997ac4c770aee2a9401. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.2:37873: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:02.216720 32711 consensus_peers.cc:485] T 5cf58a111e3048449303ec45ac091b42 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Couldn't send request to peer 16440e365430405fb10366ab70e2b429. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:16:02.219558 30175 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:16:02.231992 30200 consensus_peers.cc:485] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:16:02.323200 30199 consensus_peers.cc:485] T 2bf8c34ee16d4e0fbc6817d63b2ba52e P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:16:02.333912 30200 consensus_peers.cc:485] T 8f5e6ff35b994d46a1e1fe216eb356ce P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:16:02.406730 32709 consensus_peers.cc:485] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:16:02.424167 30199 consensus_peers.cc:485] T f694fefb42d84318a6d91532105b3c90 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:16:02.511862 371 consensus_peers.cc:485] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:16:02.564757 30200 consensus_peers.cc:485] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:16:02.600245 30199 consensus_peers.cc:485] T 58e1aaecf47944569296fc1e0780efb5 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 02:16:02.609004 17541 raft_consensus.cc:482] T 7527cc5931e24fa180efedbf81438efd P a9094240860e43bfb129193f3bfd5be6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 16440e365430405fb10366ab70e2b429)
I20240503 02:16:02.609416 17523 raft_consensus.cc:482] T 20019d63ee3e45418dc9c99bd25b72af P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 16440e365430405fb10366ab70e2b429)
I20240503 02:16:02.609279 17541 raft_consensus.cc:504] T 7527cc5931e24fa180efedbf81438efd P a9094240860e43bfb129193f3bfd5be6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } }
I20240503 02:16:02.610129 17541 leader_election.cc:288] T 7527cc5931e24fa180efedbf81438efd P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:16:02.610852 17520 raft_consensus.cc:482] T 9102ab9545c54e82b642673795cb747a P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 16440e365430405fb10366ab70e2b429)
I20240503 02:16:02.610989 17520 raft_consensus.cc:504] T 9102ab9545c54e82b642673795cb747a P a9094240860e43bfb129193f3bfd5be6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:16:02.611624 17520 leader_election.cc:288] T 9102ab9545c54e82b642673795cb747a P a9094240860e43bfb129193f3bfd5be6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:16:02.612426 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 30054
W20240503 02:16:02.691463 370 connection.cc:535] client connection to 127.22.108.3:42679 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:16:02.691596 32708 connection.cc:535] client connection to 127.22.108.3:42679 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:16:02.692225 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 30184
W20240503 02:16:02.693011 32708 consensus_peers.cc:485] T c87e7cbed0e843f5b5bb9bddef598cdb P 8a666e798fe8477d8dd1e51b385d5ead -> Peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Couldn't send request to peer a9094240860e43bfb129193f3bfd5be6. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.3:42679: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:02.693236 32708 consensus_peers.cc:485] T e165c71f8b6645a6936b58b2a8da1842 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Couldn't send request to peer a9094240860e43bfb129193f3bfd5be6. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.3:42679: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:02.693397 32708 consensus_peers.cc:485] T ebcd2c6e1722442fbe603cfcccb56ab0 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Couldn't send request to peer a9094240860e43bfb129193f3bfd5be6. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.3:42679: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:02.693533 370 consensus_peers.cc:485] T 83e8dad806564b128d5f0fe559d8ffe1 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Couldn't send request to peer a9094240860e43bfb129193f3bfd5be6. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.3:42679: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:02.693655 370 consensus_peers.cc:485] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 -> Peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Couldn't send request to peer a9094240860e43bfb129193f3bfd5be6. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.3:42679: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:02.693742 370 consensus_peers.cc:485] T 08e4d119d90f4146ba53a1b18efe2386 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Couldn't send request to peer a9094240860e43bfb129193f3bfd5be6. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.3:42679: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:02.693852 370 consensus_peers.cc:485] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 -> Peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Couldn't send request to peer a9094240860e43bfb129193f3bfd5be6. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.3:42679: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:02.694283 30199 consensus_peers.cc:485] T 96de1c26ae884ea18b63b328db040ba1 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Couldn't send request to peer a9094240860e43bfb129193f3bfd5be6. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.3:42679: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:02.694411 30199 consensus_peers.cc:485] T 1c9200fa969a46e6a48c64293801382b P 1f10f80f13c84e80b03eda7085a3b012 -> Peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Couldn't send request to peer a9094240860e43bfb129193f3bfd5be6. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.3:42679: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:02.694512 30199 consensus_peers.cc:485] T 9fa3f1e452c24c7d83a36f10284b8b5d P 1f10f80f13c84e80b03eda7085a3b012 -> Peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Couldn't send request to peer a9094240860e43bfb129193f3bfd5be6. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.3:42679: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:02.694586 30199 consensus_peers.cc:485] T 5c46d27e89ae412a9f18c27f18b68347 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Couldn't send request to peer a9094240860e43bfb129193f3bfd5be6. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.3:42679: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:02.694684 30199 consensus_peers.cc:485] T c19017f829a74cc2b6dc41a2a25a3831 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Couldn't send request to peer a9094240860e43bfb129193f3bfd5be6. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.3:42679: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:02.694763 30199 consensus_peers.cc:485] T b90b887127f44c0d920225be10077c64 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Couldn't send request to peer a9094240860e43bfb129193f3bfd5be6. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.3:42679: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:02.694919 30199 consensus_peers.cc:485] T b76b807d2eb24f3ead50bdc0bb1b0231 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Couldn't send request to peer a9094240860e43bfb129193f3bfd5be6. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.3:42679: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:02.694986 30199 consensus_peers.cc:485] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Couldn't send request to peer a9094240860e43bfb129193f3bfd5be6. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.3:42679: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:02.697746 371 consensus_peers.cc:485] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Couldn't send request to peer 9a4d92bc25374fb3960fde7d10be309d. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 02:16:02.700532 30306 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:16:02.701238 30199 consensus_peers.cc:462] T 6bf84757d1fb44689da830a82e857787 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:16:02.747888 32708 consensus_peers.cc:485] T dafe0f997f23405ca0a1bc6c0f7eab42 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 02:16:02.766635 17439 raft_consensus.cc:482] T 04e47da8cf7445dcacaf1f7c8950bd34 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 155735c7f8b14997ac4c770aee2a9401)
I20240503 02:16:02.766866 17439 raft_consensus.cc:504] T 04e47da8cf7445dcacaf1f7c8950bd34 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:16:02.767793 17439 leader_election.cc:288] T 04e47da8cf7445dcacaf1f7c8950bd34 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:16:02.769313 371 leader_election.cc:334] T 04e47da8cf7445dcacaf1f7c8950bd34 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Network error: Client connection negotiation failed: client connection to 127.22.108.2:37873: connect: Connection refused (error 111)
W20240503 02:16:02.769508 371 leader_election.cc:334] T 04e47da8cf7445dcacaf1f7c8950bd34 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:16:02.769609 371 leader_election.cc:302] T 04e47da8cf7445dcacaf1f7c8950bd34 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 155735c7f8b14997ac4c770aee2a9401, 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:16:02.769865 17439 raft_consensus.cc:2726] T 04e47da8cf7445dcacaf1f7c8950bd34 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:16:02.793227 17439 raft_consensus.cc:482] T a52bee1fdc9d48b1a3463ecd62c8b44e P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 155735c7f8b14997ac4c770aee2a9401)
I20240503 02:16:02.793422 17439 raft_consensus.cc:504] T a52bee1fdc9d48b1a3463ecd62c8b44e P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:16:02.794050 17439 leader_election.cc:288] T a52bee1fdc9d48b1a3463ecd62c8b44e P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
W20240503 02:16:02.795414 370 leader_election.cc:334] T a52bee1fdc9d48b1a3463ecd62c8b44e P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Network error: Client connection negotiation failed: client connection to 127.22.108.3:42679: connect: Connection refused (error 111)
W20240503 02:16:02.795576 371 leader_election.cc:334] T a52bee1fdc9d48b1a3463ecd62c8b44e P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Network error: Client connection negotiation failed: client connection to 127.22.108.2:37873: connect: Connection refused (error 111)
I20240503 02:16:02.795703 371 leader_election.cc:302] T a52bee1fdc9d48b1a3463ecd62c8b44e P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 155735c7f8b14997ac4c770aee2a9401, a9094240860e43bfb129193f3bfd5be6
I20240503 02:16:02.796109 17439 raft_consensus.cc:2726] T a52bee1fdc9d48b1a3463ecd62c8b44e P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:16:02.797415 32708 consensus_peers.cc:485] T eb70e12d4fdb4d15a93d37dfdc659a14 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 02:16:02.815982 17449 raft_consensus.cc:482] T b7a2201f9605454ea12df4e3a7e5a63e P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 155735c7f8b14997ac4c770aee2a9401)
I20240503 02:16:02.816172 17449 raft_consensus.cc:504] T b7a2201f9605454ea12df4e3a7e5a63e P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:16:02.816810 17449 leader_election.cc:288] T b7a2201f9605454ea12df4e3a7e5a63e P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
W20240503 02:16:02.818285 32709 leader_election.cc:334] T b7a2201f9605454ea12df4e3a7e5a63e P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Network error: Client connection negotiation failed: client connection to 127.22.108.2:37873: connect: Connection refused (error 111)
W20240503 02:16:02.818574 32711 leader_election.cc:334] T b7a2201f9605454ea12df4e3a7e5a63e P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111)
I20240503 02:16:02.818759 32711 leader_election.cc:302] T b7a2201f9605454ea12df4e3a7e5a63e P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead; no voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429
I20240503 02:16:02.819200 17449 raft_consensus.cc:2726] T b7a2201f9605454ea12df4e3a7e5a63e P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:16:02.842849 32708 consensus_peers.cc:485] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:16:02.866086 32708 consensus_peers.cc:485] T ac4d497de05c49edbd6e0edac73c058d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:16:02.917611 370 consensus_peers.cc:485] T 48b76d7755294d37860842a4a73ab28b P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 02:16:02.968469 17509 raft_consensus.cc:482] T 35a5a7f4fe184b9fadc94c98b61deaf0 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 155735c7f8b14997ac4c770aee2a9401)
I20240503 02:16:02.968678 17509 raft_consensus.cc:504] T 35a5a7f4fe184b9fadc94c98b61deaf0 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:16:02.969465 17509 leader_election.cc:288] T 35a5a7f4fe184b9fadc94c98b61deaf0 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:16:02.970786 373 leader_election.cc:334] T 35a5a7f4fe184b9fadc94c98b61deaf0 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111)
W20240503 02:16:02.971061 371 leader_election.cc:334] T 35a5a7f4fe184b9fadc94c98b61deaf0 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Network error: Client connection negotiation failed: client connection to 127.22.108.2:37873: connect: Connection refused (error 111)
I20240503 02:16:02.971154 371 leader_election.cc:302] T 35a5a7f4fe184b9fadc94c98b61deaf0 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429
I20240503 02:16:02.971410 17509 raft_consensus.cc:2726] T 35a5a7f4fe184b9fadc94c98b61deaf0 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:16:02.971406 371 leader_election.cc:334] T 35a5a7f4fe184b9fadc94c98b61deaf0 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:16:02.989522 17449 raft_consensus.cc:482] T 51f5f7297e6448d0b0e9afeedad01429 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 155735c7f8b14997ac4c770aee2a9401)
I20240503 02:16:02.989701 17449 raft_consensus.cc:504] T 51f5f7297e6448d0b0e9afeedad01429 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:16:02.990440 17449 leader_election.cc:288] T 51f5f7297e6448d0b0e9afeedad01429 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
W20240503 02:16:02.991690 32709 leader_election.cc:334] T 51f5f7297e6448d0b0e9afeedad01429 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Network error: Client connection negotiation failed: client connection to 127.22.108.2:37873: connect: Connection refused (error 111)
I20240503 02:16:03.083058 17524 raft_consensus.cc:482] T 51f5f7297e6448d0b0e9afeedad01429 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 155735c7f8b14997ac4c770aee2a9401)
I20240503 02:16:03.083250 17524 raft_consensus.cc:504] T 51f5f7297e6448d0b0e9afeedad01429 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } }
I20240503 02:16:03.084163 17524 leader_election.cc:288] T 51f5f7297e6448d0b0e9afeedad01429 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:16:03.084728 17524 raft_consensus.cc:482] T 08ee37b160d441ddabb640be907d6771 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 155735c7f8b14997ac4c770aee2a9401)
I20240503 02:16:03.084897 17524 raft_consensus.cc:504] T 08ee37b160d441ddabb640be907d6771 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:16:03.086112 17524 leader_election.cc:288] T 08ee37b160d441ddabb640be907d6771 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:16:03.090070 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e47da8cf7445dcacaf1f7c8950bd34" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "1f10f80f13c84e80b03eda7085a3b012" is_pre_election: true
I20240503 02:16:03.091161 17449 raft_consensus.cc:482] T 08ee37b160d441ddabb640be907d6771 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 155735c7f8b14997ac4c770aee2a9401)
I20240503 02:16:03.091380 17449 raft_consensus.cc:504] T 08ee37b160d441ddabb640be907d6771 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:16:03.093014 17449 leader_election.cc:288] T 08ee37b160d441ddabb640be907d6771 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
W20240503 02:16:03.093554 30199 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111) [suppressed 172 similar messages]
I20240503 02:16:03.096524 17551 raft_consensus.cc:482] T 6635aacafed54994965ad48b2df4f163 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 155735c7f8b14997ac4c770aee2a9401)
W20240503 02:16:03.096746 32709 leader_election.cc:334] T 08ee37b160d441ddabb640be907d6771 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Network error: Client connection negotiation failed: client connection to 127.22.108.2:37873: connect: Connection refused (error 111)
I20240503 02:16:03.096762 17551 raft_consensus.cc:504] T 6635aacafed54994965ad48b2df4f163 P 1f10f80f13c84e80b03eda7085a3b012 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:16:03.098088 17551 leader_election.cc:288] T 6635aacafed54994965ad48b2df4f163 P 1f10f80f13c84e80b03eda7085a3b012 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
W20240503 02:16:03.100953 370 consensus_peers.cc:485] T 52a541b1d3db4228bc5a27af521f0b93 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:16:03.101109 30199 consensus_peers.cc:485] T 5faccb32e55d4cb689ff491fd20884e6 P 1f10f80f13c84e80b03eda7085a3b012 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 02:16:03.103880 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 30184
I20240503 02:16:03.107394 17509 raft_consensus.cc:482] T e61579e74a1a4a9dbd9a09f9b34d9c8d P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 155735c7f8b14997ac4c770aee2a9401)
I20240503 02:16:03.107534 17509 raft_consensus.cc:504] T e61579e74a1a4a9dbd9a09f9b34d9c8d P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:16:03.108078 17509 leader_election.cc:288] T e61579e74a1a4a9dbd9a09f9b34d9c8d P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
W20240503 02:16:03.109561 371 leader_election.cc:334] T e61579e74a1a4a9dbd9a09f9b34d9c8d P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Network error: Client connection negotiation failed: client connection to 127.22.108.2:37873: connect: Connection refused (error 111)
W20240503 02:16:03.109763 373 leader_election.cc:334] T e61579e74a1a4a9dbd9a09f9b34d9c8d P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111)
I20240503 02:16:03.109874 373 leader_election.cc:302] T e61579e74a1a4a9dbd9a09f9b34d9c8d P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429
I20240503 02:16:03.110157 17509 raft_consensus.cc:2726] T e61579e74a1a4a9dbd9a09f9b34d9c8d P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:16:03.110316 17439 raft_consensus.cc:482] T 20019d63ee3e45418dc9c99bd25b72af P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 16440e365430405fb10366ab70e2b429)
I20240503 02:16:03.110456 17439 raft_consensus.cc:504] T 20019d63ee3e45418dc9c99bd25b72af P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:16:03.111003 17439 leader_election.cc:288] T 20019d63ee3e45418dc9c99bd25b72af P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
W20240503 02:16:03.112375 373 leader_election.cc:334] T 20019d63ee3e45418dc9c99bd25b72af P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111)
W20240503 02:16:03.112514 370 leader_election.cc:334] T 20019d63ee3e45418dc9c99bd25b72af P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Network error: Client connection negotiation failed: client connection to 127.22.108.3:42679: connect: Connection refused (error 111)
I20240503 02:16:03.112679 370 leader_election.cc:302] T 20019d63ee3e45418dc9c99bd25b72af P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 16440e365430405fb10366ab70e2b429, a9094240860e43bfb129193f3bfd5be6
I20240503 02:16:03.112932 17439 raft_consensus.cc:2726] T 20019d63ee3e45418dc9c99bd25b72af P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:16:03.113835 17439 raft_consensus.cc:482] T 32a2dbf80daf40d6b21442a39f62f14c P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 16440e365430405fb10366ab70e2b429)
I20240503 02:16:03.113962 17439 raft_consensus.cc:504] T 32a2dbf80daf40d6b21442a39f62f14c P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:16:03.114543 17439 leader_election.cc:288] T 32a2dbf80daf40d6b21442a39f62f14c P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975)
I20240503 02:16:03.115355 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a2dbf80daf40d6b21442a39f62f14c" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
W20240503 02:16:03.115721 373 leader_election.cc:334] T 32a2dbf80daf40d6b21442a39f62f14c P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111)
I20240503 02:16:03.115985 371 leader_election.cc:302] T 32a2dbf80daf40d6b21442a39f62f14c P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 16440e365430405fb10366ab70e2b429, 8a666e798fe8477d8dd1e51b385d5ead
I20240503 02:16:03.116209 17439 raft_consensus.cc:2726] T 32a2dbf80daf40d6b21442a39f62f14c P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:16:03.123070 17439 raft_consensus.cc:482] T 63b416f4e004490b8e38a96fd065dfea P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 155735c7f8b14997ac4c770aee2a9401)
I20240503 02:16:03.123191 17439 raft_consensus.cc:504] T 63b416f4e004490b8e38a96fd065dfea P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:16:03.123699 17439 leader_election.cc:288] T 63b416f4e004490b8e38a96fd065dfea P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
W20240503 02:16:03.124837 373 leader_election.cc:334] T 63b416f4e004490b8e38a96fd065dfea P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111)
W20240503 02:16:03.125065 371 leader_election.cc:334] T 63b416f4e004490b8e38a96fd065dfea P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Network error: Client connection negotiation failed: client connection to 127.22.108.2:37873: connect: Connection refused (error 111)
I20240503 02:16:03.125169 371 leader_election.cc:302] T 63b416f4e004490b8e38a96fd065dfea P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429
I20240503 02:16:03.125387 17439 raft_consensus.cc:2726] T 63b416f4e004490b8e38a96fd065dfea P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:16:03.132397 17449 raft_consensus.cc:482] T 32a2dbf80daf40d6b21442a39f62f14c P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 16440e365430405fb10366ab70e2b429)
I20240503 02:16:03.132625 17449 raft_consensus.cc:504] T 32a2dbf80daf40d6b21442a39f62f14c P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:16:03.133479 17449 leader_election.cc:288] T 32a2dbf80daf40d6b21442a39f62f14c P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:16:03.134078 434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a2dbf80daf40d6b21442a39f62f14c" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:16:03.134310 434 raft_consensus.cc:2454] T 32a2dbf80daf40d6b21442a39f62f14c P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
W20240503 02:16:03.134708 32711 leader_election.cc:334] T 32a2dbf80daf40d6b21442a39f62f14c P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111)
I20240503 02:16:03.134872 32709 leader_election.cc:302] T 32a2dbf80daf40d6b21442a39f62f14c P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, fbbc2be6382e47908ecb7c68c367bd26; no voters: 16440e365430405fb10366ab70e2b429
I20240503 02:16:03.135305 17449 raft_consensus.cc:2781] T 32a2dbf80daf40d6b21442a39f62f14c P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:16:03.135401 17449 raft_consensus.cc:482] T 32a2dbf80daf40d6b21442a39f62f14c P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting leader election (detected failure of leader 16440e365430405fb10366ab70e2b429)
I20240503 02:16:03.135499 17449 raft_consensus.cc:3037] T 32a2dbf80daf40d6b21442a39f62f14c P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:03.138676 17449 raft_consensus.cc:504] T 32a2dbf80daf40d6b21442a39f62f14c P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:16:03.139796 434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a2dbf80daf40d6b21442a39f62f14c" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:16:03.139994 434 raft_consensus.cc:3037] T 32a2dbf80daf40d6b21442a39f62f14c P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:03.139413 17449 leader_election.cc:288] T 32a2dbf80daf40d6b21442a39f62f14c P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 election: Requested vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
W20240503 02:16:03.140719 32711 leader_election.cc:334] T 32a2dbf80daf40d6b21442a39f62f14c P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111)
W20240503 02:16:03.142632 370 consensus_peers.cc:485] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 02:16:03.143050 434 raft_consensus.cc:2454] T 32a2dbf80daf40d6b21442a39f62f14c P fbbc2be6382e47908ecb7c68c367bd26 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 2.
I20240503 02:16:03.143446 32709 leader_election.cc:302] T 32a2dbf80daf40d6b21442a39f62f14c P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, fbbc2be6382e47908ecb7c68c367bd26; no voters: 16440e365430405fb10366ab70e2b429
I20240503 02:16:03.143710 17449 raft_consensus.cc:2781] T 32a2dbf80daf40d6b21442a39f62f14c P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:16:03.144135 17449 raft_consensus.cc:686] T 32a2dbf80daf40d6b21442a39f62f14c P 8a666e798fe8477d8dd1e51b385d5ead [term 2 LEADER]: Becoming Leader. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Running, Role: LEADER
I20240503 02:16:03.144927 17557 raft_consensus.cc:482] T 5f4b72b138c64ac58e6477cd2b1fb215 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 16440e365430405fb10366ab70e2b429)
I20240503 02:16:03.144822 17449 consensus_queue.cc:237] T 32a2dbf80daf40d6b21442a39f62f14c P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:16:03.145103 17557 raft_consensus.cc:504] T 5f4b72b138c64ac58e6477cd2b1fb215 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:16:03.145779 17439 raft_consensus.cc:482] T 16cf978ac35341d4adea71a17a91194c P fbbc2be6382e47908ecb7c68c367bd26 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 16440e365430405fb10366ab70e2b429)
I20240503 02:16:03.145973 17439 raft_consensus.cc:504] T 16cf978ac35341d4adea71a17a91194c P fbbc2be6382e47908ecb7c68c367bd26 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:16:03.146080 17509 raft_consensus.cc:482] T c97c7a843366426887115e13f35faf52 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 16440e365430405fb10366ab70e2b429)
I20240503 02:16:03.146225 17509 raft_consensus.cc:504] T c97c7a843366426887115e13f35faf52 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:16:03.146596 17439 leader_election.cc:288] T 16cf978ac35341d4adea71a17a91194c P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:16:03.146677 17509 leader_election.cc:288] T c97c7a843366426887115e13f35faf52 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
W20240503 02:16:03.148730 373 leader_election.cc:334] T 16cf978ac35341d4adea71a17a91194c P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111)
W20240503 02:16:03.148906 373 leader_election.cc:334] T c97c7a843366426887115e13f35faf52 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111)
I20240503 02:16:03.149889 17558 raft_consensus.cc:482] T c97c7a843366426887115e13f35faf52 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 16440e365430405fb10366ab70e2b429)
I20240503 02:16:03.150342 305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97c7a843366426887115e13f35faf52" candidate_uuid: "fbbc2be6382e47908ecb7c68c367bd26" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8a666e798fe8477d8dd1e51b385d5ead" is_pre_election: true
I20240503 02:16:03.150671 17509 raft_consensus.cc:482] T df37d88053e2419fbe40d03023d0c756 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 16440e365430405fb10366ab70e2b429)
I20240503 02:16:03.150822 17509 raft_consensus.cc:504] T df37d88053e2419fbe40d03023d0c756 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
W20240503 02:16:03.151134 371 leader_election.cc:334] T 16cf978ac35341d4adea71a17a91194c P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:16:03.151257 371 leader_election.cc:302] T 16cf978ac35341d4adea71a17a91194c P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 16440e365430405fb10366ab70e2b429, 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:16:03.151412 17509 leader_election.cc:288] T df37d88053e2419fbe40d03023d0c756 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
I20240503 02:16:03.151537 17509 raft_consensus.cc:2726] T 16cf978ac35341d4adea71a17a91194c P fbbc2be6382e47908ecb7c68c367bd26 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:16:03.151942 17439 raft_consensus.cc:482] T 28f3f05f0a344cd683cb9e010273f908 P fbbc2be6382e47908ecb7c68c367bd26 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 16440e365430405fb10366ab70e2b429)
I20240503 02:16:03.152143 17439 raft_consensus.cc:504] T 28f3f05f0a344cd683cb9e010273f908 P fbbc2be6382e47908ecb7c68c367bd26 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } }
I20240503 02:16:03.150182 17558 raft_consensus.cc:504] T c97c7a843366426887115e13f35faf52 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:16:03.153052 305 raft_consensus.cc:2454] T c97c7a843366426887115e13f35faf52 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbbc2be6382e47908ecb7c68c367bd26 in term 1.
I20240503 02:16:03.153432 17558 leader_election.cc:288] T c97c7a843366426887115e13f35faf52 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
I20240503 02:16:03.154228 434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97c7a843366426887115e13f35faf52" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26" is_pre_election: true
I20240503 02:16:03.154414 434 raft_consensus.cc:2454] T c97c7a843366426887115e13f35faf52 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 1.
I20240503 02:16:03.154615 29730 catalog_manager.cc:5484] T 32a2dbf80daf40d6b21442a39f62f14c P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: term changed from 1 to 2, leader changed from 16440e365430405fb10366ab70e2b429 (127.22.108.1) to 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6). New cstate: current_term: 2 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
W20240503 02:16:03.154917 32711 leader_election.cc:334] T 5f4b72b138c64ac58e6477cd2b1fb215 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111)
I20240503 02:16:03.155014 32709 leader_election.cc:302] T c97c7a843366426887115e13f35faf52 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, fbbc2be6382e47908ecb7c68c367bd26; no voters:
I20240503 02:16:03.155586 17449 raft_consensus.cc:2781] T c97c7a843366426887115e13f35faf52 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:16:03.155678 17449 raft_consensus.cc:482] T c97c7a843366426887115e13f35faf52 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting leader election (detected failure of leader 16440e365430405fb10366ab70e2b429)
I20240503 02:16:03.155766 17449 raft_consensus.cc:3037] T c97c7a843366426887115e13f35faf52 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:03.156312 17439 leader_election.cc:288] T 28f3f05f0a344cd683cb9e010273f908 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
W20240503 02:16:03.156320 32711 leader_election.cc:334] T c97c7a843366426887115e13f35faf52 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111)
W20240503 02:16:03.158162 373 leader_election.cc:334] T df37d88053e2419fbe40d03023d0c756 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111)
W20240503 02:16:03.158288 373 leader_election.cc:334] T 28f3f05f0a344cd683cb9e010273f908 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111)
I20240503 02:16:03.158954 17449 raft_consensus.cc:504] T c97c7a843366426887115e13f35faf52 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:16:03.159636 17449 leader_election.cc:288] T c97c7a843366426887115e13f35faf52 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 election: Requested vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), fbbc2be6382e47908ecb7c68c367bd26 (127.22.108.7:44241)
W20240503 02:16:03.160482 32709 leader_election.cc:334] T 5f4b72b138c64ac58e6477cd2b1fb215 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Network error: Client connection negotiation failed: client connection to 127.22.108.2:37873: connect: Connection refused (error 111)
I20240503 02:16:03.160610 32709 leader_election.cc:302] T 5f4b72b138c64ac58e6477cd2b1fb215 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead; no voters: 155735c7f8b14997ac4c770aee2a9401, 16440e365430405fb10366ab70e2b429
W20240503 02:16:03.160791 32711 leader_election.cc:334] T c97c7a843366426887115e13f35faf52 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111)
I20240503 02:16:03.160898 434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97c7a843366426887115e13f35faf52" candidate_uuid: "8a666e798fe8477d8dd1e51b385d5ead" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "fbbc2be6382e47908ecb7c68c367bd26"
I20240503 02:16:03.161139 434 raft_consensus.cc:3037] T c97c7a843366426887115e13f35faf52 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:03.161597 17558 raft_consensus.cc:2726] T 5f4b72b138c64ac58e6477cd2b1fb215 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:16:03.162040 17557 leader_election.cc:288] T 5f4b72b138c64ac58e6477cd2b1fb215 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873)
I20240503 02:16:03.163753 371 leader_election.cc:302] T c97c7a843366426887115e13f35faf52 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, fbbc2be6382e47908ecb7c68c367bd26; no voters: 16440e365430405fb10366ab70e2b429
I20240503 02:16:03.163986 434 raft_consensus.cc:2454] T c97c7a843366426887115e13f35faf52 P fbbc2be6382e47908ecb7c68c367bd26 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a666e798fe8477d8dd1e51b385d5ead in term 2.
I20240503 02:16:03.164124 17439 raft_consensus.cc:2741] T c97c7a843366426887115e13f35faf52 P fbbc2be6382e47908ecb7c68c367bd26 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20240503 02:16:03.164361 32709 leader_election.cc:302] T c97c7a843366426887115e13f35faf52 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead, fbbc2be6382e47908ecb7c68c367bd26; no voters: 16440e365430405fb10366ab70e2b429
I20240503 02:16:03.164867 17557 raft_consensus.cc:2781] T c97c7a843366426887115e13f35faf52 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:16:03.165041 17557 raft_consensus.cc:686] T c97c7a843366426887115e13f35faf52 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 LEADER]: Becoming Leader. State: Replica: 8a666e798fe8477d8dd1e51b385d5ead, State: Running, Role: LEADER
I20240503 02:16:03.165305 17557 consensus_queue.cc:237] T c97c7a843366426887115e13f35faf52 P 8a666e798fe8477d8dd1e51b385d5ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
W20240503 02:16:03.168205 371 leader_election.cc:334] T 28f3f05f0a344cd683cb9e010273f908 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:16:03.168359 371 leader_election.cc:302] T 28f3f05f0a344cd683cb9e010273f908 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 16440e365430405fb10366ab70e2b429, 9a4d92bc25374fb3960fde7d10be309d
I20240503 02:16:03.168640 17439 raft_consensus.cc:2726] T 28f3f05f0a344cd683cb9e010273f908 P fbbc2be6382e47908ecb7c68c367bd26 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:16:03.179386 17439 raft_consensus.cc:482] T e8a76caac93d441b92e65380bf6dc03e P fbbc2be6382e47908ecb7c68c367bd26 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a9094240860e43bfb129193f3bfd5be6)
I20240503 02:16:03.179553 17439 raft_consensus.cc:504] T e8a76caac93d441b92e65380bf6dc03e P fbbc2be6382e47908ecb7c68c367bd26 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:16:03.180156 17439 leader_election.cc:288] T e8a76caac93d441b92e65380bf6dc03e P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:16:03.182543 29730 catalog_manager.cc:5484] T c97c7a843366426887115e13f35faf52 P 8a666e798fe8477d8dd1e51b385d5ead reported cstate change: term changed from 1 to 2, leader changed from 16440e365430405fb10366ab70e2b429 (127.22.108.1) to 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6). New cstate: current_term: 2 leader_uuid: "8a666e798fe8477d8dd1e51b385d5ead" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } }
W20240503 02:16:03.182806 371 connection.cc:535] client connection to 127.22.108.4:33935 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:16:03.183197 32709 connection.cc:535] client connection to 127.22.108.4:33935 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:16:03.184450 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 32692
W20240503 02:16:03.184609 32709 consensus_peers.cc:485] T ac4a753bca5f44058199942f05ef181d P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): Couldn't send request to peer 1f10f80f13c84e80b03eda7085a3b012. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.4:33935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:03.184739 32709 leader_election.cc:334] T 51f5f7297e6448d0b0e9afeedad01429 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): Network error: Client connection negotiation failed: client connection to 127.22.108.4:33935: connect: Connection refused (error 111)
I20240503 02:16:03.184854 32709 leader_election.cc:302] T 51f5f7297e6448d0b0e9afeedad01429 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead; no voters: 155735c7f8b14997ac4c770aee2a9401, 1f10f80f13c84e80b03eda7085a3b012
I20240503 02:16:03.185138 17557 raft_consensus.cc:2726] T 51f5f7297e6448d0b0e9afeedad01429 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:16:03.185220 32709 consensus_peers.cc:485] T 00aedc4902a745d6a859cbbf99529a7f P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): Couldn't send request to peer 1f10f80f13c84e80b03eda7085a3b012. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.4:33935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:03.185322 32709 consensus_peers.cc:485] T 5800022ae50043ee8b1af1e134e86e6a P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): Couldn't send request to peer 1f10f80f13c84e80b03eda7085a3b012. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.4:33935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:03.185431 32709 leader_election.cc:334] T 08ee37b160d441ddabb640be907d6771 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): Network error: Client connection negotiation failed: client connection to 127.22.108.4:33935: connect: Connection refused (error 111)
W20240503 02:16:03.185438 370 leader_election.cc:334] T e8a76caac93d441b92e65380bf6dc03e P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Network error: Client connection negotiation failed: client connection to 127.22.108.3:42679: connect: Connection refused (error 111)
I20240503 02:16:03.185528 32709 leader_election.cc:302] T 08ee37b160d441ddabb640be907d6771 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a666e798fe8477d8dd1e51b385d5ead; no voters: 155735c7f8b14997ac4c770aee2a9401, 1f10f80f13c84e80b03eda7085a3b012
W20240503 02:16:03.185837 32709 consensus_peers.cc:485] T 3f3ec15b1ec043d1a8f252f42a584f63 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): Couldn't send request to peer 1f10f80f13c84e80b03eda7085a3b012. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.4:33935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:03.185946 32709 consensus_peers.cc:485] T 3c437d9d466348f2863c407666c348aa P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): Couldn't send request to peer 1f10f80f13c84e80b03eda7085a3b012. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.4:33935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:03.186045 32709 consensus_peers.cc:485] T e165c71f8b6645a6936b58b2a8da1842 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): Couldn't send request to peer 1f10f80f13c84e80b03eda7085a3b012. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.4:33935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:03.186147 32709 consensus_peers.cc:485] T 368d683f40cd4d0994e7bbe90e045d16 P 8a666e798fe8477d8dd1e51b385d5ead -> Peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): Couldn't send request to peer 1f10f80f13c84e80b03eda7085a3b012. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.4:33935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:16:03.186533 17557 raft_consensus.cc:2726] T 08ee37b160d441ddabb640be907d6771 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:16:03.186825 371 leader_election.cc:334] T e8a76caac93d441b92e65380bf6dc03e P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595): Network error: Client connection negotiation failed: client connection to 127.22.108.5:35595: connect: Connection refused (error 111)
I20240503 02:16:03.186944 371 leader_election.cc:302] T e8a76caac93d441b92e65380bf6dc03e P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 9a4d92bc25374fb3960fde7d10be309d, a9094240860e43bfb129193f3bfd5be6
I20240503 02:16:03.187206 17439 raft_consensus.cc:2726] T e8a76caac93d441b92e65380bf6dc03e P fbbc2be6382e47908ecb7c68c367bd26 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:16:03.187281 371 leader_election.cc:334] T df37d88053e2419fbe40d03023d0c756 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): Network error: Client connection negotiation failed: client connection to 127.22.108.4:33935: connect: Connection refused (error 111)
I20240503 02:16:03.187381 371 leader_election.cc:302] T df37d88053e2419fbe40d03023d0c756 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 16440e365430405fb10366ab70e2b429, 1f10f80f13c84e80b03eda7085a3b012
W20240503 02:16:03.187644 371 consensus_peers.cc:485] T 4cc7861a21f94dabb01133bac9ba0b46 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): Couldn't send request to peer 1f10f80f13c84e80b03eda7085a3b012. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.4:33935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:03.187744 371 consensus_peers.cc:485] T cf0743dee35b411e8fd069322d4f3da6 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): Couldn't send request to peer 1f10f80f13c84e80b03eda7085a3b012. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.4:33935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:16:03.187914 17439 raft_consensus.cc:2726] T df37d88053e2419fbe40d03023d0c756 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:16:03.194257 346 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:16:03.202971 370 consensus_peers.cc:485] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:16:03.248231 370 consensus_peers.cc:485] T 737a39a4f65b4e9ab2ce65c0d7f62dcb P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:16:03.269744 371 consensus_peers.cc:485] T 88f696d03330448d879ed7b60364c28c P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): Couldn't send request to peer 1f10f80f13c84e80b03eda7085a3b012. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.4:33935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:03.289393 371 consensus_peers.cc:485] T 9c6e204df01e450598b05c3429574700 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): Couldn't send request to peer 1f10f80f13c84e80b03eda7085a3b012. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.4:33935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:16:03.337320 17439 raft_consensus.cc:482] T 5a5d5658fe5e4104a15b404cb2022b7c P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a9094240860e43bfb129193f3bfd5be6)
I20240503 02:16:03.337517 17439 raft_consensus.cc:504] T 5a5d5658fe5e4104a15b404cb2022b7c P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:16:03.338145 17439 leader_election.cc:288] T 5a5d5658fe5e4104a15b404cb2022b7c P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
W20240503 02:16:03.339308 370 leader_election.cc:334] T 5a5d5658fe5e4104a15b404cb2022b7c P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Network error: Client connection negotiation failed: client connection to 127.22.108.3:42679: connect: Connection refused (error 111)
W20240503 02:16:03.339458 371 leader_election.cc:334] T 5a5d5658fe5e4104a15b404cb2022b7c P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873): Network error: Client connection negotiation failed: client connection to 127.22.108.2:37873: connect: Connection refused (error 111)
I20240503 02:16:03.339587 371 leader_election.cc:302] T 5a5d5658fe5e4104a15b404cb2022b7c P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 155735c7f8b14997ac4c770aee2a9401, a9094240860e43bfb129193f3bfd5be6
I20240503 02:16:03.339836 17439 raft_consensus.cc:2726] T 5a5d5658fe5e4104a15b404cb2022b7c P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:16:03.372360 371 consensus_peers.cc:485] T b7b284cffcb04ce883a24f6a8796748f P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): Couldn't send request to peer 1f10f80f13c84e80b03eda7085a3b012. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.4:33935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:03.392125 371 consensus_peers.cc:485] T 83c7d1060ccf49a2adda4302600a9876 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): Couldn't send request to peer 1f10f80f13c84e80b03eda7085a3b012. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.4:33935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:16:03.404743 17439 raft_consensus.cc:482] T b6c9fd3cbbd5453897aa072600db569d P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a9094240860e43bfb129193f3bfd5be6)
I20240503 02:16:03.404944 17439 raft_consensus.cc:504] T b6c9fd3cbbd5453897aa072600db569d P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:16:03.405629 17439 leader_election.cc:288] T b6c9fd3cbbd5453897aa072600db569d P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719)
W20240503 02:16:03.407368 370 leader_election.cc:334] T b6c9fd3cbbd5453897aa072600db569d P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Network error: Client connection negotiation failed: client connection to 127.22.108.3:42679: connect: Connection refused (error 111)
W20240503 02:16:03.408182 373 leader_election.cc:334] T b6c9fd3cbbd5453897aa072600db569d P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719): Network error: Client connection negotiation failed: client connection to 127.22.108.1:36719: connect: Connection refused (error 111)
I20240503 02:16:03.408330 373 leader_election.cc:302] T b6c9fd3cbbd5453897aa072600db569d P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: 16440e365430405fb10366ab70e2b429, a9094240860e43bfb129193f3bfd5be6
I20240503 02:16:03.408665 17439 raft_consensus.cc:2726] T b6c9fd3cbbd5453897aa072600db569d P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:16:03.487821 371 consensus_peers.cc:485] T 7263a253c5884bce987d28e188ccca11 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935): Couldn't send request to peer 1f10f80f13c84e80b03eda7085a3b012. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.4:33935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:16:03.498521 17439 raft_consensus.cc:482] T 88b57085de3144cea3f64922ae5ccf55 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a9094240860e43bfb129193f3bfd5be6)
I20240503 02:16:03.498754 17439 raft_consensus.cc:504] T 88b57085de3144cea3f64922ae5ccf55 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:16:03.499429 17439 leader_election.cc:288] T 88b57085de3144cea3f64922ae5ccf55 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
W20240503 02:16:03.500829 370 leader_election.cc:334] T 88b57085de3144cea3f64922ae5ccf55 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111)
W20240503 02:16:03.501020 370 leader_election.cc:334] T 88b57085de3144cea3f64922ae5ccf55 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679): Network error: Client connection negotiation failed: client connection to 127.22.108.3:42679: connect: Connection refused (error 111)
I20240503 02:16:03.501150 370 leader_election.cc:302] T 88b57085de3144cea3f64922ae5ccf55 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbbc2be6382e47908ecb7c68c367bd26; no voters: a9094240860e43bfb129193f3bfd5be6, f575ce92c0224eeb80b9fae5f6561fbe
I20240503 02:16:03.501397 17439 raft_consensus.cc:2726] T 88b57085de3144cea3f64922ae5ccf55 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:16:03.548789 17558 raft_consensus.cc:482] T 6635aacafed54994965ad48b2df4f163 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 155735c7f8b14997ac4c770aee2a9401)
I20240503 02:16:03.549106 17558 raft_consensus.cc:504] T 6635aacafed54994965ad48b2df4f163 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } }
I20240503 02:16:03.550186 17558 leader_election.cc:288] T 6635aacafed54994965ad48b2df4f163 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:16:03.550379 17558 raft_consensus.cc:482] T db0756f0f331402aaef0583e1f9e5318 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a9094240860e43bfb129193f3bfd5be6)
I20240503 02:16:03.550539 17558 raft_consensus.cc:504] T db0756f0f331402aaef0583e1f9e5318 P 8a666e798fe8477d8dd1e51b385d5ead [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:16:03.551087 17557 raft_consensus.cc:482] T 88509fbaa62645c58b7ab1a432eb8727 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a9094240860e43bfb129193f3bfd5be6)
I20240503 02:16:03.551261 17558 leader_election.cc:288] T db0756f0f331402aaef0583e1f9e5318 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:16:03.551234 17557 raft_consensus.cc:504] T 88509fbaa62645c58b7ab1a432eb8727 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:16:03.551688 17449 raft_consensus.cc:482] T d2a8d96585fd4b4881d1bd028b1a9d36 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a9094240860e43bfb129193f3bfd5be6)
I20240503 02:16:03.551784 17557 leader_election.cc:288] T 88509fbaa62645c58b7ab1a432eb8727 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 16440e365430405fb10366ab70e2b429 (127.22.108.1:36719), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:16:03.551859 17449 raft_consensus.cc:504] T d2a8d96585fd4b4881d1bd028b1a9d36 P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "1f10f80f13c84e80b03eda7085a3b012" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 33935 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:16:03.552677 17449 leader_election.cc:288] T d2a8d96585fd4b4881d1bd028b1a9d36 P 8a666e798fe8477d8dd1e51b385d5ead [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), 1f10f80f13c84e80b03eda7085a3b012 (127.22.108.4:33935)
I20240503 02:16:03.553400 17569 raft_consensus.cc:482] T e80b43971dc246d9bfd5e1c603f8415c P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a9094240860e43bfb129193f3bfd5be6)
I20240503 02:16:03.553550 17569 raft_consensus.cc:504] T e80b43971dc246d9bfd5e1c603f8415c P 8a666e798fe8477d8dd1e51b385d5ead [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16440e365430405fb10366ab70e2b429" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 36719 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "8a666e798fe8477d8dd1e51b385d5ead" member_type: VOTER last_known_addr { host: "127.22.108.6" port: 39975 } }
I20240503 02:16:03.554286 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 32692
W20240503 02:16:03.614970 371 connection.cc:535] client connection to 127.22.108.6:39975 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:16:03.615970 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 355
W20240503 02:16:03.616325 371 consensus_peers.cc:485] T 3351f8900c7446179da1fbe65f835e0a P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): Couldn't send request to peer 8a666e798fe8477d8dd1e51b385d5ead. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.6:39975: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:03.616468 371 consensus_peers.cc:485] T 9c6e204df01e450598b05c3429574700 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): Couldn't send request to peer 8a666e798fe8477d8dd1e51b385d5ead. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.6:39975: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:03.616573 371 consensus_peers.cc:485] T 5ccde7688c4e49b5b9d89d6e048c308e P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): Couldn't send request to peer 8a666e798fe8477d8dd1e51b385d5ead. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.6:39975: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:03.616644 371 consensus_peers.cc:485] T f82e505ac13a41019e422b9dc1e4fef9 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): Couldn't send request to peer 8a666e798fe8477d8dd1e51b385d5ead. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.6:39975: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:03.616734 371 consensus_peers.cc:485] T 2baf70e0e1894cea9fdeb92279bc0cff P fbbc2be6382e47908ecb7c68c367bd26 -> Peer 8a666e798fe8477d8dd1e51b385d5ead (127.22.108.6:39975): Couldn't send request to peer 8a666e798fe8477d8dd1e51b385d5ead. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.6:39975: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:03.618589 370 consensus_peers.cc:485] T 62d5e9ea7af8435bb34028df2b3f1457 P fbbc2be6382e47908ecb7c68c367bd26 -> Peer f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233): Couldn't send request to peer f575ce92c0224eeb80b9fae5f6561fbe. Status: Network error: Client connection negotiation failed: client connection to 127.22.108.8:42233: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 02:16:03.622337 476 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:16:03.968456 17509 raft_consensus.cc:482] T c5cd64614c53487ea1c6cc4a4ca5c379 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a9094240860e43bfb129193f3bfd5be6)
I20240503 02:16:03.968761 17509 raft_consensus.cc:504] T c5cd64614c53487ea1c6cc4a4ca5c379 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:16:03.969789 17509 leader_election.cc:288] T c5cd64614c53487ea1c6cc4a4ca5c379 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595)
I20240503 02:16:03.969974 17509 raft_consensus.cc:482] T a6505f95a1c547748fb5cb7a4918b129 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 155735c7f8b14997ac4c770aee2a9401)
I20240503 02:16:03.970136 17509 raft_consensus.cc:504] T a6505f95a1c547748fb5cb7a4918b129 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } } peers { permanent_uuid: "155735c7f8b14997ac4c770aee2a9401" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 37873 } } peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } }
I20240503 02:16:03.970831 17509 leader_election.cc:288] T a6505f95a1c547748fb5cb7a4918b129 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 155735c7f8b14997ac4c770aee2a9401 (127.22.108.2:37873), a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679)
I20240503 02:16:03.970996 17509 raft_consensus.cc:482] T b8b119c0d8044f2ab19b2100950270d6 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a9094240860e43bfb129193f3bfd5be6)
I20240503 02:16:03.971169 17509 raft_consensus.cc:504] T b8b119c0d8044f2ab19b2100950270d6 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a9094240860e43bfb129193f3bfd5be6" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 42679 } } peers { permanent_uuid: "9a4d92bc25374fb3960fde7d10be309d" member_type: VOTER last_known_addr { host: "127.22.108.5" port: 35595 } } peers { permanent_uuid: "f575ce92c0224eeb80b9fae5f6561fbe" member_type: VOTER last_known_addr { host: "127.22.108.8" port: 42233 } } peers { permanent_uuid: "fbbc2be6382e47908ecb7c68c367bd26" member_type: VOTER last_known_addr { host: "127.22.108.7" port: 44241 } }
I20240503 02:16:03.971995 17509 leader_election.cc:288] T b8b119c0d8044f2ab19b2100950270d6 P fbbc2be6382e47908ecb7c68c367bd26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a9094240860e43bfb129193f3bfd5be6 (127.22.108.3:42679), 9a4d92bc25374fb3960fde7d10be309d (127.22.108.5:35595), f575ce92c0224eeb80b9fae5f6561fbe (127.22.108.8:42233)
I20240503 02:16:03.974472 17579 raft_consensus.cc:482] T 96de1c26ae884ea18b63b328db040ba1 P fbbc2be6382e47908ecb7c68c367bd26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1f10f80f13c84e80b03eda7085a3b012)
I20240503 02:16:03.974720 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 355
I20240503 02:16:04.034353 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 29703
I20240503 02:16:04.040349 29764 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:16:04.342744 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 29703
2024-05-03T02:16:04Z chronyd exiting
I20240503 02:16:04.412788 22960 test_util.cc:175] -----------------------------------------------
I20240503 02:16:04.412880 22960 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714702278677268-22960-0
[ FAILED ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/0, where GetParam() = 4-byte object <00-00 00-00> (228094 ms)
[----------] 1 test from DDLDuringRebalancingTest (228094 ms total)
[----------] 1 test from RebalancingDuringElectionStormTest
[ RUN ] RebalancingDuringElectionStormTest.RoundRobin/0
I20240503 02:16:04.415095 22960 test_util.cc:268] Using random seed: 1295996494
I20240503 02:16:04.416967 22960 ts_itest-base.cc:112] Starting cluster with:
I20240503 02:16:04.417095 22960 ts_itest-base.cc:113] --------------
I20240503 02:16:04.417153 22960 ts_itest-base.cc:114] 8 tablet servers
I20240503 02:16:04.417230 22960 ts_itest-base.cc:115] 3 replicas per TS
I20240503 02:16:04.417281 22960 ts_itest-base.cc:116] --------------
2024-05-03T02:16:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:16:04Z Disabled control of system clock
I20240503 02:16:04.450045 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.108.62:34029
--webserver_interface=127.22.108.62
--webserver_port=0
--builtin_ntp_servers=127.22.108.20:46405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.108.62:34029
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240503 02:16:04.787963 17592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:16:04.788318 17592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:16:04.788456 17592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:16:04.796097 17592 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:16:04.796265 17592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:16:04.796362 17592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:16:04.796445 17592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:16:04.796514 17592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:16:04.806010 17592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:46405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.22.108.62:34029
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.108.62:34029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.22.108.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:16:04.807435 17592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:16:04.809278 17592 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:16:04.821388 17597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:16:04.822103 17598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:16:04.823024 17600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:16:04.823841 17592 server_base.cc:1034] running on GCE node
I20240503 02:16:04.824667 17592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:16:04.826223 17592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:16:04.827433 17592 hybrid_clock.cc:648] HybridClock initialized: now 1714702564827390 us; error 33 us; skew 500 ppm
I20240503 02:16:04.827780 17592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:16:04.830153 17592 webserver.cc:460] Webserver started at http://127.22.108.62:39083/ using document root <none> and password file <none>
I20240503 02:16:04.830919 17592 fs_manager.cc:359] Metadata directory not provided
I20240503 02:16:04.831034 17592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:16:04.831352 17592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:16:04.834054 17592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "e35aa698a7e04588b6e4d7d1c155c7ac"
format_stamp: "Formatted at 2024-05-03 02:16:04 on dist-test-slave-sc15"
I20240503 02:16:04.834825 17592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "e35aa698a7e04588b6e4d7d1c155c7ac"
format_stamp: "Formatted at 2024-05-03 02:16:04 on dist-test-slave-sc15"
I20240503 02:16:04.840237 17592 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 02:16:04.843729 17606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:16:04.845345 17592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:16:04.845530 17592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e35aa698a7e04588b6e4d7d1c155c7ac"
format_stamp: "Formatted at 2024-05-03 02:16:04 on dist-test-slave-sc15"
I20240503 02:16:04.845742 17592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:16:04.867260 17592 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:16:04.868135 17592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:16:04.868430 17592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:16:04.885886 17657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.62:34029 every 8 connection(s)
I20240503 02:16:04.885880 17592 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.62:34029
I20240503 02:16:04.887364 17592 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 02:16:04.891142 17658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:04.892369 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 17592
I20240503 02:16:04.892589 22960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 02:16:04.901579 17658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e35aa698a7e04588b6e4d7d1c155c7ac: Bootstrap starting.
I20240503 02:16:04.905067 17658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e35aa698a7e04588b6e4d7d1c155c7ac: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:04.906344 17658 log.cc:824] T 00000000000000000000000000000000 P e35aa698a7e04588b6e4d7d1c155c7ac: Log is configured to *not* fsync() on all Append() calls
I20240503 02:16:04.909014 17658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e35aa698a7e04588b6e4d7d1c155c7ac: No bootstrap required, opened a new log
I20240503 02:16:04.915407 17658 raft_consensus.cc:348] T 00000000000000000000000000000000 P e35aa698a7e04588b6e4d7d1c155c7ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35aa698a7e04588b6e4d7d1c155c7ac" member_type: VOTER last_known_addr { host: "127.22.108.62" port: 34029 } }
I20240503 02:16:04.915836 17658 raft_consensus.cc:374] T 00000000000000000000000000000000 P e35aa698a7e04588b6e4d7d1c155c7ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:04.915967 17658 raft_consensus.cc:729] T 00000000000000000000000000000000 P e35aa698a7e04588b6e4d7d1c155c7ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e35aa698a7e04588b6e4d7d1c155c7ac, State: Initialized, Role: FOLLOWER
I20240503 02:16:04.916636 17658 consensus_queue.cc:260] T 00000000000000000000000000000000 P e35aa698a7e04588b6e4d7d1c155c7ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35aa698a7e04588b6e4d7d1c155c7ac" member_type: VOTER last_known_addr { host: "127.22.108.62" port: 34029 } }
I20240503 02:16:04.916847 17658 raft_consensus.cc:388] T 00000000000000000000000000000000 P e35aa698a7e04588b6e4d7d1c155c7ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:16:04.916971 17658 raft_consensus.cc:482] T 00000000000000000000000000000000 P e35aa698a7e04588b6e4d7d1c155c7ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:16:04.917138 17658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e35aa698a7e04588b6e4d7d1c155c7ac [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:04.919196 17658 raft_consensus.cc:504] T 00000000000000000000000000000000 P e35aa698a7e04588b6e4d7d1c155c7ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35aa698a7e04588b6e4d7d1c155c7ac" member_type: VOTER last_known_addr { host: "127.22.108.62" port: 34029 } }
I20240503 02:16:04.919706 17658 leader_election.cc:302] T 00000000000000000000000000000000 P e35aa698a7e04588b6e4d7d1c155c7ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e35aa698a7e04588b6e4d7d1c155c7ac; no voters:
I20240503 02:16:04.920320 17658 leader_election.cc:288] T 00000000000000000000000000000000 P e35aa698a7e04588b6e4d7d1c155c7ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:16:04.920445 17663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e35aa698a7e04588b6e4d7d1c155c7ac [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:16:04.921998 17663 raft_consensus.cc:686] T 00000000000000000000000000000000 P e35aa698a7e04588b6e4d7d1c155c7ac [term 1 LEADER]: Becoming Leader. State: Replica: e35aa698a7e04588b6e4d7d1c155c7ac, State: Running, Role: LEADER
I20240503 02:16:04.922493 17663 consensus_queue.cc:237] T 00000000000000000000000000000000 P e35aa698a7e04588b6e4d7d1c155c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35aa698a7e04588b6e4d7d1c155c7ac" member_type: VOTER last_known_addr { host: "127.22.108.62" port: 34029 } }
I20240503 02:16:04.922600 17658 sys_catalog.cc:564] T 00000000000000000000000000000000 P e35aa698a7e04588b6e4d7d1c155c7ac [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:16:04.925536 17665 sys_catalog.cc:455] T 00000000000000000000000000000000 P e35aa698a7e04588b6e4d7d1c155c7ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader e35aa698a7e04588b6e4d7d1c155c7ac. Latest consensus state: current_term: 1 leader_uuid: "e35aa698a7e04588b6e4d7d1c155c7ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35aa698a7e04588b6e4d7d1c155c7ac" member_type: VOTER last_known_addr { host: "127.22.108.62" port: 34029 } } }
I20240503 02:16:04.925446 17664 sys_catalog.cc:455] T 00000000000000000000000000000000 P e35aa698a7e04588b6e4d7d1c155c7ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e35aa698a7e04588b6e4d7d1c155c7ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35aa698a7e04588b6e4d7d1c155c7ac" member_type: VOTER last_known_addr { host: "127.22.108.62" port: 34029 } } }
I20240503 02:16:04.925886 17665 sys_catalog.cc:458] T 00000000000000000000000000000000 P e35aa698a7e04588b6e4d7d1c155c7ac [sys.catalog]: This master's current role is: LEADER
I20240503 02:16:04.925921 17664 sys_catalog.cc:458] T 00000000000000000000000000000000 P e35aa698a7e04588b6e4d7d1c155c7ac [sys.catalog]: This master's current role is: LEADER
I20240503 02:16:04.927899 17670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:16:04.933144 17670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:16:04.941959 17670 catalog_manager.cc:1347] Generated new cluster ID: 21a88c03436e4c6096c7db1679f50f5f
I20240503 02:16:04.942112 17670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:16:04.956912 17670 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:16:04.958207 17670 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:16:04.974843 17670 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e35aa698a7e04588b6e4d7d1c155c7ac: Generated new TSK 0
I20240503 02:16:04.975807 17670 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:16:04.984438 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.1:0
--local_ip_for_outbound_sockets=127.22.108.1
--webserver_interface=127.22.108.1
--webserver_port=0
--tserver_master_addrs=127.22.108.62:34029
--builtin_ntp_servers=127.22.108.20:46405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 02:16:05.323827 17682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:16:05.324127 17682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:16:05.324265 17682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:16:05.331630 17682 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:16:05.331754 17682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:16:05.331912 17682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.1
I20240503 02:16:05.340824 17682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:46405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.22.108.1
--webserver_port=0
--tserver_master_addrs=127.22.108.62:34029
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.1
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:16:05.342242 17682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:16:05.343964 17682 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:16:05.356671 17687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:16:05.357385 17688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:16:05.358237 17690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:16:05.359134 17682 server_base.cc:1034] running on GCE node
I20240503 02:16:05.359740 17682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:16:05.360687 17682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:16:05.361894 17682 hybrid_clock.cc:648] HybridClock initialized: now 1714702565361857 us; error 28 us; skew 500 ppm
I20240503 02:16:05.362231 17682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:16:05.370751 17682 webserver.cc:460] Webserver started at http://127.22.108.1:38531/ using document root <none> and password file <none>
I20240503 02:16:05.371532 17682 fs_manager.cc:359] Metadata directory not provided
I20240503 02:16:05.371671 17682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:16:05.372006 17682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:16:05.374686 17682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "836b43a7ee9146948319b8d37434b630"
format_stamp: "Formatted at 2024-05-03 02:16:05 on dist-test-slave-sc15"
I20240503 02:16:05.375450 17682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "836b43a7ee9146948319b8d37434b630"
format_stamp: "Formatted at 2024-05-03 02:16:05 on dist-test-slave-sc15"
I20240503 02:16:05.380553 17682 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20240503 02:16:05.383888 17696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:16:05.385249 17682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:16:05.385401 17682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "836b43a7ee9146948319b8d37434b630"
format_stamp: "Formatted at 2024-05-03 02:16:05 on dist-test-slave-sc15"
I20240503 02:16:05.385587 17682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:16:05.417637 17682 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:16:05.418536 17682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:16:05.418812 17682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:16:05.419742 17682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:16:05.421315 17682 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:16:05.421422 17682 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:16:05.421536 17682 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:16:05.421607 17682 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:16:05.453713 17682 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.1:45837
I20240503 02:16:05.453778 17808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.1:45837 every 8 connection(s)
I20240503 02:16:05.455055 17682 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 02:16:05.458102 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 17682
I20240503 02:16:05.458312 22960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 02:16:05.461709 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.2:0
--local_ip_for_outbound_sockets=127.22.108.2
--webserver_interface=127.22.108.2
--webserver_port=0
--tserver_master_addrs=127.22.108.62:34029
--builtin_ntp_servers=127.22.108.20:46405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:16:05.468832 17809 heartbeater.cc:347] Connected to a master server at 127.22.108.62:34029
I20240503 02:16:05.469169 17809 heartbeater.cc:464] Registering TS with master...
I20240503 02:16:05.469894 17809 heartbeater.cc:511] Master 127.22.108.62:34029 requested a full tablet report, sending...
I20240503 02:16:05.471829 17623 ts_manager.cc:194] Registered new tserver with Master: 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837)
I20240503 02:16:05.473366 17623 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.1:60849
W20240503 02:16:05.796808 17813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:16:05.797163 17813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:16:05.797309 17813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:16:05.804700 17813 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:16:05.804852 17813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:16:05.805047 17813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.2
I20240503 02:16:05.813988 17813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:46405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.22.108.2
--webserver_port=0
--tserver_master_addrs=127.22.108.62:34029
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.2
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:16:05.815411 17813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:16:05.817312 17813 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:16:05.829419 17818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:16:05.830152 17821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:16:05.831279 17819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:16:05.832203 17813 server_base.cc:1034] running on GCE node
I20240503 02:16:05.832783 17813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:16:05.833778 17813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:16:05.834976 17813 hybrid_clock.cc:648] HybridClock initialized: now 1714702565834914 us; error 56 us; skew 500 ppm
I20240503 02:16:05.835300 17813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:16:05.837471 17813 webserver.cc:460] Webserver started at http://127.22.108.2:34381/ using document root <none> and password file <none>
I20240503 02:16:05.838174 17813 fs_manager.cc:359] Metadata directory not provided
I20240503 02:16:05.838279 17813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:16:05.838585 17813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:16:05.841243 17813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "0f3838ee7ad04e94aa65ac750dad0833"
format_stamp: "Formatted at 2024-05-03 02:16:05 on dist-test-slave-sc15"
I20240503 02:16:05.842005 17813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "0f3838ee7ad04e94aa65ac750dad0833"
format_stamp: "Formatted at 2024-05-03 02:16:05 on dist-test-slave-sc15"
I20240503 02:16:05.847024 17813 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240503 02:16:05.850327 17827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:16:05.851680 17813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:16:05.851837 17813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "0f3838ee7ad04e94aa65ac750dad0833"
format_stamp: "Formatted at 2024-05-03 02:16:05 on dist-test-slave-sc15"
I20240503 02:16:05.852026 17813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:16:05.880698 17813 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:16:05.881629 17813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:16:05.881902 17813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:16:05.882875 17813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:16:05.884503 17813 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:16:05.884604 17813 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:16:05.884716 17813 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:16:05.884788 17813 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:16:05.924521 17813 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.2:41647
I20240503 02:16:05.924572 17939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.2:41647 every 8 connection(s)
I20240503 02:16:05.925894 17813 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 02:16:05.935885 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 17813
I20240503 02:16:05.936072 22960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 02:16:05.938403 17940 heartbeater.cc:347] Connected to a master server at 127.22.108.62:34029
I20240503 02:16:05.938728 17940 heartbeater.cc:464] Registering TS with master...
I20240503 02:16:05.939527 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.3:0
--local_ip_for_outbound_sockets=127.22.108.3
--webserver_interface=127.22.108.3
--webserver_port=0
--tserver_master_addrs=127.22.108.62:34029
--builtin_ntp_servers=127.22.108.20:46405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:16:05.939651 17940 heartbeater.cc:511] Master 127.22.108.62:34029 requested a full tablet report, sending...
I20240503 02:16:05.941201 17623 ts_manager.cc:194] Registered new tserver with Master: 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647)
I20240503 02:16:05.942263 17623 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.2:51983
W20240503 02:16:06.276929 17943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:16:06.277280 17943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:16:06.277375 17943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:16:06.284915 17943 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:16:06.285084 17943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:16:06.285231 17943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.3
I20240503 02:16:06.294450 17943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:46405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.22.108.3
--webserver_port=0
--tserver_master_addrs=127.22.108.62:34029
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.3
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:16:06.295758 17943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:16:06.297577 17943 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:16:06.310034 17948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:16:06.310808 17949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:16:06.311295 17951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:16:06.312461 17943 server_base.cc:1034] running on GCE node
I20240503 02:16:06.313119 17943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:16:06.314049 17943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:16:06.315239 17943 hybrid_clock.cc:648] HybridClock initialized: now 1714702566315179 us; error 50 us; skew 500 ppm
I20240503 02:16:06.315560 17943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:16:06.317970 17943 webserver.cc:460] Webserver started at http://127.22.108.3:38041/ using document root <none> and password file <none>
I20240503 02:16:06.318675 17943 fs_manager.cc:359] Metadata directory not provided
I20240503 02:16:06.318770 17943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:16:06.319056 17943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:16:06.321605 17943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "1cd8517d70d94ae494e713df7eb61b12"
format_stamp: "Formatted at 2024-05-03 02:16:06 on dist-test-slave-sc15"
I20240503 02:16:06.322341 17943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "1cd8517d70d94ae494e713df7eb61b12"
format_stamp: "Formatted at 2024-05-03 02:16:06 on dist-test-slave-sc15"
I20240503 02:16:06.327502 17943 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 02:16:06.330799 17957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:16:06.332058 17943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 02:16:06.332201 17943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "1cd8517d70d94ae494e713df7eb61b12"
format_stamp: "Formatted at 2024-05-03 02:16:06 on dist-test-slave-sc15"
I20240503 02:16:06.332386 17943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:16:06.349547 17943 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:16:06.350519 17943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:16:06.350765 17943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:16:06.351706 17943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:16:06.353215 17943 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:16:06.353300 17943 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:16:06.353399 17943 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:16:06.353467 17943 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:16:06.384527 17943 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.3:34759
I20240503 02:16:06.384583 18069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.3:34759 every 8 connection(s)
I20240503 02:16:06.385845 17943 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:16:06.393937 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 17943
I20240503 02:16:06.394080 22960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 02:16:06.397121 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.4:0
--local_ip_for_outbound_sockets=127.22.108.4
--webserver_interface=127.22.108.4
--webserver_port=0
--tserver_master_addrs=127.22.108.62:34029
--builtin_ntp_servers=127.22.108.20:46405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:16:06.398677 18070 heartbeater.cc:347] Connected to a master server at 127.22.108.62:34029
I20240503 02:16:06.398986 18070 heartbeater.cc:464] Registering TS with master...
I20240503 02:16:06.399724 18070 heartbeater.cc:511] Master 127.22.108.62:34029 requested a full tablet report, sending...
I20240503 02:16:06.401185 17623 ts_manager.cc:194] Registered new tserver with Master: 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759)
I20240503 02:16:06.402323 17623 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.3:42987
I20240503 02:16:06.476317 17809 heartbeater.cc:503] Master 127.22.108.62:34029 was elected leader, sending a full tablet report...
W20240503 02:16:06.732328 18073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:16:06.732668 18073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:16:06.732810 18073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:16:06.740433 18073 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:16:06.740579 18073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:16:06.740737 18073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.4
I20240503 02:16:06.749869 18073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:46405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.22.108.4
--webserver_port=0
--tserver_master_addrs=127.22.108.62:34029
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.4
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:16:06.751279 18073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:16:06.753090 18073 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:16:06.766122 18081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:16:06.766753 18078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:16:06.767323 18079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:16:06.768308 18073 server_base.cc:1034] running on GCE node
I20240503 02:16:06.768914 18073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:16:06.769923 18073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:16:06.771116 18073 hybrid_clock.cc:648] HybridClock initialized: now 1714702566771059 us; error 52 us; skew 500 ppm
I20240503 02:16:06.771432 18073 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:16:06.773659 18073 webserver.cc:460] Webserver started at http://127.22.108.4:33233/ using document root <none> and password file <none>
I20240503 02:16:06.774375 18073 fs_manager.cc:359] Metadata directory not provided
I20240503 02:16:06.774483 18073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:16:06.774801 18073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:16:06.777429 18073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "4325d7d0ed484ad78a6ee54e9a003431"
format_stamp: "Formatted at 2024-05-03 02:16:06 on dist-test-slave-sc15"
I20240503 02:16:06.778195 18073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "4325d7d0ed484ad78a6ee54e9a003431"
format_stamp: "Formatted at 2024-05-03 02:16:06 on dist-test-slave-sc15"
I20240503 02:16:06.783329 18073 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 02:16:06.786720 18087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:16:06.788054 18073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 02:16:06.788211 18073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "4325d7d0ed484ad78a6ee54e9a003431"
format_stamp: "Formatted at 2024-05-03 02:16:06 on dist-test-slave-sc15"
I20240503 02:16:06.788408 18073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:16:06.809397 18073 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:16:06.810330 18073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:16:06.810597 18073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:16:06.811550 18073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:16:06.813061 18073 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:16:06.813159 18073 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:16:06.813277 18073 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:16:06.813349 18073 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:16:06.849264 18073 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.4:39703
I20240503 02:16:06.849313 18199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.4:39703 every 8 connection(s)
I20240503 02:16:06.850664 18073 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:16:06.851092 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 18073
I20240503 02:16:06.851296 22960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 02:16:06.854660 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.5:0
--local_ip_for_outbound_sockets=127.22.108.5
--webserver_interface=127.22.108.5
--webserver_port=0
--tserver_master_addrs=127.22.108.62:34029
--builtin_ntp_servers=127.22.108.20:46405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:16:06.864248 18200 heartbeater.cc:347] Connected to a master server at 127.22.108.62:34029
I20240503 02:16:06.864555 18200 heartbeater.cc:464] Registering TS with master...
I20240503 02:16:06.865365 18200 heartbeater.cc:511] Master 127.22.108.62:34029 requested a full tablet report, sending...
I20240503 02:16:06.866688 17623 ts_manager.cc:194] Registered new tserver with Master: 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703)
I20240503 02:16:06.867465 17623 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.4:36475
I20240503 02:16:06.945326 17940 heartbeater.cc:503] Master 127.22.108.62:34029 was elected leader, sending a full tablet report...
W20240503 02:16:07.187755 18203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:16:07.188097 18203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:16:07.188221 18203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:16:07.195863 18203 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:16:07.196013 18203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:16:07.196187 18203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.5
I20240503 02:16:07.205242 18203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:46405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.108.5
--webserver_port=0
--tserver_master_addrs=127.22.108.62:34029
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.5
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:16:07.206663 18203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:16:07.208510 18203 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:16:07.221591 18208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:16:07.222127 18209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:16:07.223256 18211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:16:07.224261 18203 server_base.cc:1034] running on GCE node
I20240503 02:16:07.224900 18203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:16:07.225939 18203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:16:07.227137 18203 hybrid_clock.cc:648] HybridClock initialized: now 1714702567227084 us; error 49 us; skew 500 ppm
I20240503 02:16:07.227509 18203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:16:07.229964 18203 webserver.cc:460] Webserver started at http://127.22.108.5:41917/ using document root <none> and password file <none>
I20240503 02:16:07.230751 18203 fs_manager.cc:359] Metadata directory not provided
I20240503 02:16:07.230857 18203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:16:07.231196 18203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:16:07.233961 18203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "7a727cc21a924056b804ff5460b06d9a"
format_stamp: "Formatted at 2024-05-03 02:16:07 on dist-test-slave-sc15"
I20240503 02:16:07.234771 18203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "7a727cc21a924056b804ff5460b06d9a"
format_stamp: "Formatted at 2024-05-03 02:16:07 on dist-test-slave-sc15"
I20240503 02:16:07.240339 18203 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 02:16:07.243837 18217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:16:07.245258 18203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:16:07.245405 18203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "7a727cc21a924056b804ff5460b06d9a"
format_stamp: "Formatted at 2024-05-03 02:16:07 on dist-test-slave-sc15"
I20240503 02:16:07.245601 18203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:16:07.271119 18203 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:16:07.272010 18203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:16:07.272290 18203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:16:07.273334 18203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:16:07.274886 18203 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:16:07.274973 18203 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:16:07.275086 18203 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:16:07.275156 18203 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:16:07.306656 18203 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.5:37187
I20240503 02:16:07.306756 18329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.5:37187 every 8 connection(s)
I20240503 02:16:07.308004 18203 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:16:07.309048 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 18203
I20240503 02:16:07.309229 22960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 02:16:07.312908 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.6:0
--local_ip_for_outbound_sockets=127.22.108.6
--webserver_interface=127.22.108.6
--webserver_port=0
--tserver_master_addrs=127.22.108.62:34029
--builtin_ntp_servers=127.22.108.20:46405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:16:07.324043 18330 heartbeater.cc:347] Connected to a master server at 127.22.108.62:34029
I20240503 02:16:07.324371 18330 heartbeater.cc:464] Registering TS with master...
I20240503 02:16:07.325296 18330 heartbeater.cc:511] Master 127.22.108.62:34029 requested a full tablet report, sending...
I20240503 02:16:07.326762 17623 ts_manager.cc:194] Registered new tserver with Master: 7a727cc21a924056b804ff5460b06d9a (127.22.108.5:37187)
I20240503 02:16:07.327536 17623 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.5:58559
I20240503 02:16:07.404909 18070 heartbeater.cc:503] Master 127.22.108.62:34029 was elected leader, sending a full tablet report...
W20240503 02:16:07.654551 18333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:16:07.654871 18333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:16:07.655078 18333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:16:07.663005 18333 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:16:07.663128 18333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:16:07.663281 18333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.6
I20240503 02:16:07.672367 18333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:46405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.22.108.6
--webserver_port=0
--tserver_master_addrs=127.22.108.62:34029
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.6
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:16:07.673764 18333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:16:07.675505 18333 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:16:07.687768 18339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:16:07.688300 18338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:16:07.689358 18341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:16:07.690333 18333 server_base.cc:1034] running on GCE node
I20240503 02:16:07.690917 18333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:16:07.691805 18333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:16:07.693008 18333 hybrid_clock.cc:648] HybridClock initialized: now 1714702567692958 us; error 39 us; skew 500 ppm
I20240503 02:16:07.693401 18333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:16:07.695541 18333 webserver.cc:460] Webserver started at http://127.22.108.6:45105/ using document root <none> and password file <none>
I20240503 02:16:07.696264 18333 fs_manager.cc:359] Metadata directory not provided
I20240503 02:16:07.696368 18333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:16:07.696676 18333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:16:07.699231 18333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "314f3c1b901a4cfdb0ed6298b9b874f2"
format_stamp: "Formatted at 2024-05-03 02:16:07 on dist-test-slave-sc15"
I20240503 02:16:07.699978 18333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "314f3c1b901a4cfdb0ed6298b9b874f2"
format_stamp: "Formatted at 2024-05-03 02:16:07 on dist-test-slave-sc15"
I20240503 02:16:07.704895 18333 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20240503 02:16:07.708165 18347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:16:07.709435 18333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:16:07.709584 18333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "314f3c1b901a4cfdb0ed6298b9b874f2"
format_stamp: "Formatted at 2024-05-03 02:16:07 on dist-test-slave-sc15"
I20240503 02:16:07.709772 18333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:16:07.742650 18333 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:16:07.743572 18333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:16:07.743865 18333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:16:07.744868 18333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:16:07.746397 18333 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:16:07.746496 18333 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:16:07.746613 18333 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:16:07.746685 18333 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:16:07.777931 18333 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.6:43647
I20240503 02:16:07.777987 18459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.6:43647 every 8 connection(s)
I20240503 02:16:07.779259 18333 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 02:16:07.779743 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 18333
I20240503 02:16:07.779933 22960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 02:16:07.783665 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.7:0
--local_ip_for_outbound_sockets=127.22.108.7
--webserver_interface=127.22.108.7
--webserver_port=0
--tserver_master_addrs=127.22.108.62:34029
--builtin_ntp_servers=127.22.108.20:46405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:16:07.793072 18460 heartbeater.cc:347] Connected to a master server at 127.22.108.62:34029
I20240503 02:16:07.793433 18460 heartbeater.cc:464] Registering TS with master...
I20240503 02:16:07.794221 18460 heartbeater.cc:511] Master 127.22.108.62:34029 requested a full tablet report, sending...
I20240503 02:16:07.795625 17623 ts_manager.cc:194] Registered new tserver with Master: 314f3c1b901a4cfdb0ed6298b9b874f2 (127.22.108.6:43647)
I20240503 02:16:07.796424 17623 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.6:39943
I20240503 02:16:07.869961 18200 heartbeater.cc:503] Master 127.22.108.62:34029 was elected leader, sending a full tablet report...
W20240503 02:16:08.133108 18463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:16:08.133437 18463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:16:08.133584 18463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:16:08.141301 18463 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:16:08.141448 18463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:16:08.141618 18463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.7
I20240503 02:16:08.151072 18463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:46405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.22.108.7
--webserver_port=0
--tserver_master_addrs=127.22.108.62:34029
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.7
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:16:08.152433 18463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:16:08.154247 18463 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:16:08.166718 18468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:16:08.167698 18469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:16:08.168445 18471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:16:08.169371 18463 server_base.cc:1034] running on GCE node
I20240503 02:16:08.169986 18463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:16:08.170943 18463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:16:08.172137 18463 hybrid_clock.cc:648] HybridClock initialized: now 1714702568172084 us; error 55 us; skew 500 ppm
I20240503 02:16:08.172443 18463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:16:08.174805 18463 webserver.cc:460] Webserver started at http://127.22.108.7:41063/ using document root <none> and password file <none>
I20240503 02:16:08.175558 18463 fs_manager.cc:359] Metadata directory not provided
I20240503 02:16:08.175674 18463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:16:08.175993 18463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:16:08.178644 18463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "eceaab49d4184b68a38415bd66b8641a"
format_stamp: "Formatted at 2024-05-03 02:16:08 on dist-test-slave-sc15"
I20240503 02:16:08.179447 18463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "eceaab49d4184b68a38415bd66b8641a"
format_stamp: "Formatted at 2024-05-03 02:16:08 on dist-test-slave-sc15"
I20240503 02:16:08.184712 18463 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20240503 02:16:08.188138 18477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:16:08.189635 18463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:16:08.189793 18463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "eceaab49d4184b68a38415bd66b8641a"
format_stamp: "Formatted at 2024-05-03 02:16:08 on dist-test-slave-sc15"
I20240503 02:16:08.189992 18463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:16:08.208474 18463 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:16:08.209390 18463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:16:08.209661 18463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:16:08.210598 18463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:16:08.212126 18463 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:16:08.212222 18463 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:16:08.212342 18463 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:16:08.212412 18463 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:16:08.243916 18463 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.7:35849
I20240503 02:16:08.243964 18589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.7:35849 every 8 connection(s)
I20240503 02:16:08.245313 18463 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 02:16:08.248231 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 18463
I20240503 02:16:08.248409 22960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 02:16:08.252156 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.8:0
--local_ip_for_outbound_sockets=127.22.108.8
--webserver_interface=127.22.108.8
--webserver_port=0
--tserver_master_addrs=127.22.108.62:34029
--builtin_ntp_servers=127.22.108.20:46405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:16:08.260331 18590 heartbeater.cc:347] Connected to a master server at 127.22.108.62:34029
I20240503 02:16:08.260712 18590 heartbeater.cc:464] Registering TS with master...
I20240503 02:16:08.261564 18590 heartbeater.cc:511] Master 127.22.108.62:34029 requested a full tablet report, sending...
I20240503 02:16:08.263113 17623 ts_manager.cc:194] Registered new tserver with Master: eceaab49d4184b68a38415bd66b8641a (127.22.108.7:35849)
I20240503 02:16:08.263936 17623 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.7:47385
I20240503 02:16:08.330437 18330 heartbeater.cc:503] Master 127.22.108.62:34029 was elected leader, sending a full tablet report...
W20240503 02:16:08.597339 18593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:16:08.597627 18593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:16:08.597750 18593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:16:08.605365 18593 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:16:08.605490 18593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:16:08.605654 18593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.8
I20240503 02:16:08.614892 18593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:46405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.22.108.8
--webserver_port=0
--tserver_master_addrs=127.22.108.62:34029
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.8
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:16:08.616299 18593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:16:08.618207 18593 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:16:08.630683 18598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:16:08.631341 18601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:16:08.632247 18599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:16:08.633431 18593 server_base.cc:1034] running on GCE node
I20240503 02:16:08.634085 18593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:16:08.635027 18593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:16:08.636246 18593 hybrid_clock.cc:648] HybridClock initialized: now 1714702568636198 us; error 44 us; skew 500 ppm
I20240503 02:16:08.636565 18593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:16:08.639137 18593 webserver.cc:460] Webserver started at http://127.22.108.8:41063/ using document root <none> and password file <none>
I20240503 02:16:08.639871 18593 fs_manager.cc:359] Metadata directory not provided
I20240503 02:16:08.639986 18593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:16:08.640308 18593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:16:08.642948 18593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "e9c890f17c8e49cf9f411caeb4df18c6"
format_stamp: "Formatted at 2024-05-03 02:16:08 on dist-test-slave-sc15"
I20240503 02:16:08.643754 18593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "e9c890f17c8e49cf9f411caeb4df18c6"
format_stamp: "Formatted at 2024-05-03 02:16:08 on dist-test-slave-sc15"
I20240503 02:16:08.649087 18593 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240503 02:16:08.652901 18607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:16:08.654388 18593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:16:08.654557 18593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "e9c890f17c8e49cf9f411caeb4df18c6"
format_stamp: "Formatted at 2024-05-03 02:16:08 on dist-test-slave-sc15"
I20240503 02:16:08.654767 18593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:16:08.671837 18593 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:16:08.672722 18593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:16:08.673012 18593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:16:08.674007 18593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:16:08.675488 18593 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:16:08.675588 18593 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:16:08.675701 18593 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:16:08.675772 18593 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:16:08.707607 18593 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.8:32871
I20240503 02:16:08.707651 18719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.8:32871 every 8 connection(s)
I20240503 02:16:08.708976 18593 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 02:16:08.717216 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 18593
I20240503 02:16:08.717406 22960 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 02:16:08.721743 18720 heartbeater.cc:347] Connected to a master server at 127.22.108.62:34029
I20240503 02:16:08.722010 18720 heartbeater.cc:464] Registering TS with master...
I20240503 02:16:08.722715 18720 heartbeater.cc:511] Master 127.22.108.62:34029 requested a full tablet report, sending...
I20240503 02:16:08.724062 17623 ts_manager.cc:194] Registered new tserver with Master: e9c890f17c8e49cf9f411caeb4df18c6 (127.22.108.8:32871)
I20240503 02:16:08.724839 17623 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.8:43345
I20240503 02:16:08.733345 22960 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 02:16:08.749655 22960 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 02:16:08.749770 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 18203
I20240503 02:16:08.757535 18325 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:16:08.798945 18460 heartbeater.cc:503] Master 127.22.108.62:34029 was elected leader, sending a full tablet report...
I20240503 02:16:08.839747 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 18203
I20240503 02:16:08.856246 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 18333
I20240503 02:16:08.864326 18455 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:16:08.944700 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 18333
I20240503 02:16:08.961753 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 18463
I20240503 02:16:08.969686 18585 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:16:09.048857 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 18463
I20240503 02:16:09.065626 22960 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1Onrea/build/asan/bin/kudu pid 18593
I20240503 02:16:09.073697 18715 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:16:09.155548 22960 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1Onrea/build/asan/bin/kudu with pid 18593
I20240503 02:16:12.923892 17623 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44922:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:16:12.965813 18134 tablet_service.cc:1467] Processing CreateTablet for tablet 6ccff572af764f51938873b46b6622d9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ffbab065274e487b9849ec0717e0c82a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:12.966322 18005 tablet_service.cc:1467] Processing CreateTablet for tablet 628548a443e64c6fae75070970bba761 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ffbab065274e487b9849ec0717e0c82a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:12.967418 18134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ccff572af764f51938873b46b6622d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:12.967667 18005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 628548a443e64c6fae75070970bba761. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:12.970223 17742 tablet_service.cc:1467] Processing CreateTablet for tablet 6ccff572af764f51938873b46b6622d9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ffbab065274e487b9849ec0717e0c82a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:12.970907 18135 tablet_service.cc:1467] Processing CreateTablet for tablet 628548a443e64c6fae75070970bba761 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ffbab065274e487b9849ec0717e0c82a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:12.971103 17742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ccff572af764f51938873b46b6622d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:12.971374 18135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 628548a443e64c6fae75070970bba761. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:12.971450 17874 tablet_service.cc:1467] Processing CreateTablet for tablet 6ccff572af764f51938873b46b6622d9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ffbab065274e487b9849ec0717e0c82a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:12.971964 18004 tablet_service.cc:1467] Processing CreateTablet for tablet c5812cc63a1f4aa2b8efdcabb1bf5b29 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ffbab065274e487b9849ec0717e0c82a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:12.972399 18004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5812cc63a1f4aa2b8efdcabb1bf5b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:12.972625 17874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ccff572af764f51938873b46b6622d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:12.973405 17744 tablet_service.cc:1467] Processing CreateTablet for tablet 628548a443e64c6fae75070970bba761 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ffbab065274e487b9849ec0717e0c82a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:12.973856 17744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 628548a443e64c6fae75070970bba761. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:12.973770 17875 tablet_service.cc:1467] Processing CreateTablet for tablet c5812cc63a1f4aa2b8efdcabb1bf5b29 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ffbab065274e487b9849ec0717e0c82a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:12.974246 17875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5812cc63a1f4aa2b8efdcabb1bf5b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:12.969856 17743 tablet_service.cc:1467] Processing CreateTablet for tablet c5812cc63a1f4aa2b8efdcabb1bf5b29 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ffbab065274e487b9849ec0717e0c82a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:12.975108 17743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5812cc63a1f4aa2b8efdcabb1bf5b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:12.996048 18749 tablet_bootstrap.cc:492] T 6ccff572af764f51938873b46b6622d9 P 4325d7d0ed484ad78a6ee54e9a003431: Bootstrap starting.
I20240503 02:16:12.999301 18751 tablet_bootstrap.cc:492] T 628548a443e64c6fae75070970bba761 P 1cd8517d70d94ae494e713df7eb61b12: Bootstrap starting.
I20240503 02:16:12.999933 18749 tablet_bootstrap.cc:654] T 6ccff572af764f51938873b46b6622d9 P 4325d7d0ed484ad78a6ee54e9a003431: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:13.000448 18750 tablet_bootstrap.cc:492] T 628548a443e64c6fae75070970bba761 P 836b43a7ee9146948319b8d37434b630: Bootstrap starting.
I20240503 02:16:13.001323 18749 log.cc:824] T 6ccff572af764f51938873b46b6622d9 P 4325d7d0ed484ad78a6ee54e9a003431: Log is configured to *not* fsync() on all Append() calls
I20240503 02:16:13.004215 18751 tablet_bootstrap.cc:654] T 628548a443e64c6fae75070970bba761 P 1cd8517d70d94ae494e713df7eb61b12: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:13.005518 18751 log.cc:824] T 628548a443e64c6fae75070970bba761 P 1cd8517d70d94ae494e713df7eb61b12: Log is configured to *not* fsync() on all Append() calls
I20240503 02:16:13.006044 18749 tablet_bootstrap.cc:492] T 6ccff572af764f51938873b46b6622d9 P 4325d7d0ed484ad78a6ee54e9a003431: No bootstrap required, opened a new log
I20240503 02:16:13.006451 18749 ts_tablet_manager.cc:1367] T 6ccff572af764f51938873b46b6622d9 P 4325d7d0ed484ad78a6ee54e9a003431: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.006s
I20240503 02:16:13.009349 18750 tablet_bootstrap.cc:654] T 628548a443e64c6fae75070970bba761 P 836b43a7ee9146948319b8d37434b630: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:13.009698 18752 tablet_bootstrap.cc:492] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 0f3838ee7ad04e94aa65ac750dad0833: Bootstrap starting.
I20240503 02:16:13.010609 18750 log.cc:824] T 628548a443e64c6fae75070970bba761 P 836b43a7ee9146948319b8d37434b630: Log is configured to *not* fsync() on all Append() calls
I20240503 02:16:13.014416 18752 tablet_bootstrap.cc:654] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 0f3838ee7ad04e94aa65ac750dad0833: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:13.016058 18752 log.cc:824] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 0f3838ee7ad04e94aa65ac750dad0833: Log is configured to *not* fsync() on all Append() calls
I20240503 02:16:13.018743 18751 tablet_bootstrap.cc:492] T 628548a443e64c6fae75070970bba761 P 1cd8517d70d94ae494e713df7eb61b12: No bootstrap required, opened a new log
I20240503 02:16:13.019111 18751 ts_tablet_manager.cc:1367] T 628548a443e64c6fae75070970bba761 P 1cd8517d70d94ae494e713df7eb61b12: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.006s
I20240503 02:16:13.019208 18749 raft_consensus.cc:348] T 6ccff572af764f51938873b46b6622d9 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:13.019675 18749 raft_consensus.cc:374] T 6ccff572af764f51938873b46b6622d9 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:13.019790 18749 raft_consensus.cc:729] T 6ccff572af764f51938873b46b6622d9 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4325d7d0ed484ad78a6ee54e9a003431, State: Initialized, Role: FOLLOWER
I20240503 02:16:13.020468 18750 tablet_bootstrap.cc:492] T 628548a443e64c6fae75070970bba761 P 836b43a7ee9146948319b8d37434b630: No bootstrap required, opened a new log
I20240503 02:16:13.020502 18749 consensus_queue.cc:260] T 6ccff572af764f51938873b46b6622d9 P 4325d7d0ed484ad78a6ee54e9a003431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:13.020705 18750 ts_tablet_manager.cc:1367] T 628548a443e64c6fae75070970bba761 P 836b43a7ee9146948319b8d37434b630: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.011s
I20240503 02:16:13.022842 18752 tablet_bootstrap.cc:492] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 0f3838ee7ad04e94aa65ac750dad0833: No bootstrap required, opened a new log
I20240503 02:16:13.023164 18752 ts_tablet_manager.cc:1367] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 0f3838ee7ad04e94aa65ac750dad0833: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.004s
I20240503 02:16:13.028528 18749 ts_tablet_manager.cc:1398] T 6ccff572af764f51938873b46b6622d9 P 4325d7d0ed484ad78a6ee54e9a003431: Time spent starting tablet: real 0.022s user 0.011s sys 0.008s
I20240503 02:16:13.029996 18751 raft_consensus.cc:348] T 628548a443e64c6fae75070970bba761 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:13.030292 18750 raft_consensus.cc:348] T 628548a443e64c6fae75070970bba761 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:13.030648 18751 raft_consensus.cc:374] T 628548a443e64c6fae75070970bba761 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:13.030822 18750 raft_consensus.cc:374] T 628548a443e64c6fae75070970bba761 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:13.030815 18751 raft_consensus.cc:729] T 628548a443e64c6fae75070970bba761 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Initialized, Role: FOLLOWER
I20240503 02:16:13.030964 18750 raft_consensus.cc:729] T 628548a443e64c6fae75070970bba761 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Initialized, Role: FOLLOWER
I20240503 02:16:13.031662 18751 consensus_queue.cc:260] T 628548a443e64c6fae75070970bba761 P 1cd8517d70d94ae494e713df7eb61b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:13.031713 18750 consensus_queue.cc:260] T 628548a443e64c6fae75070970bba761 P 836b43a7ee9146948319b8d37434b630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:13.032094 18752 raft_consensus.cc:348] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:13.033093 18752 raft_consensus.cc:374] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:13.034096 18752 raft_consensus.cc:729] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Initialized, Role: FOLLOWER
I20240503 02:16:13.034978 18752 consensus_queue.cc:260] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:13.036639 18749 tablet_bootstrap.cc:492] T 628548a443e64c6fae75070970bba761 P 4325d7d0ed484ad78a6ee54e9a003431: Bootstrap starting.
I20240503 02:16:13.037300 18751 ts_tablet_manager.cc:1398] T 628548a443e64c6fae75070970bba761 P 1cd8517d70d94ae494e713df7eb61b12: Time spent starting tablet: real 0.018s user 0.004s sys 0.012s
I20240503 02:16:13.038638 18749 tablet_bootstrap.cc:654] T 628548a443e64c6fae75070970bba761 P 4325d7d0ed484ad78a6ee54e9a003431: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:13.040117 18751 tablet_bootstrap.cc:492] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 1cd8517d70d94ae494e713df7eb61b12: Bootstrap starting.
I20240503 02:16:13.042426 18751 tablet_bootstrap.cc:654] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 1cd8517d70d94ae494e713df7eb61b12: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:13.045274 18750 ts_tablet_manager.cc:1398] T 628548a443e64c6fae75070970bba761 P 836b43a7ee9146948319b8d37434b630: Time spent starting tablet: real 0.024s user 0.021s sys 0.000s
I20240503 02:16:13.045746 18750 tablet_bootstrap.cc:492] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 836b43a7ee9146948319b8d37434b630: Bootstrap starting.
I20240503 02:16:13.046329 18752 ts_tablet_manager.cc:1398] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 0f3838ee7ad04e94aa65ac750dad0833: Time spent starting tablet: real 0.023s user 0.013s sys 0.007s
I20240503 02:16:13.046804 18752 tablet_bootstrap.cc:492] T 6ccff572af764f51938873b46b6622d9 P 0f3838ee7ad04e94aa65ac750dad0833: Bootstrap starting.
I20240503 02:16:13.047592 18750 tablet_bootstrap.cc:654] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 836b43a7ee9146948319b8d37434b630: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:13.048498 18752 tablet_bootstrap.cc:654] T 6ccff572af764f51938873b46b6622d9 P 0f3838ee7ad04e94aa65ac750dad0833: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:13.055938 18750 tablet_bootstrap.cc:492] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 836b43a7ee9146948319b8d37434b630: No bootstrap required, opened a new log
I20240503 02:16:13.056301 18750 ts_tablet_manager.cc:1367] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 836b43a7ee9146948319b8d37434b630: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20240503 02:16:13.057765 18750 raft_consensus.cc:348] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:13.058156 18750 raft_consensus.cc:374] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:13.058298 18750 raft_consensus.cc:729] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Initialized, Role: FOLLOWER
I20240503 02:16:13.058674 18750 consensus_queue.cc:260] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 836b43a7ee9146948319b8d37434b630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:13.059486 18750 ts_tablet_manager.cc:1398] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 836b43a7ee9146948319b8d37434b630: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:16:13.059911 18750 tablet_bootstrap.cc:492] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630: Bootstrap starting.
I20240503 02:16:13.061264 18752 tablet_bootstrap.cc:492] T 6ccff572af764f51938873b46b6622d9 P 0f3838ee7ad04e94aa65ac750dad0833: No bootstrap required, opened a new log
I20240503 02:16:13.061462 18752 ts_tablet_manager.cc:1367] T 6ccff572af764f51938873b46b6622d9 P 0f3838ee7ad04e94aa65ac750dad0833: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.004s
I20240503 02:16:13.062244 18750 tablet_bootstrap.cc:654] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:13.062505 18752 raft_consensus.cc:348] T 6ccff572af764f51938873b46b6622d9 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:13.062776 18752 raft_consensus.cc:374] T 6ccff572af764f51938873b46b6622d9 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:13.062919 18752 raft_consensus.cc:729] T 6ccff572af764f51938873b46b6622d9 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Initialized, Role: FOLLOWER
I20240503 02:16:13.063205 18752 consensus_queue.cc:260] T 6ccff572af764f51938873b46b6622d9 P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:13.063892 18752 ts_tablet_manager.cc:1398] T 6ccff572af764f51938873b46b6622d9 P 0f3838ee7ad04e94aa65ac750dad0833: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:16:13.066465 18751 tablet_bootstrap.cc:492] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 1cd8517d70d94ae494e713df7eb61b12: No bootstrap required, opened a new log
I20240503 02:16:13.066634 18751 ts_tablet_manager.cc:1367] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 1cd8517d70d94ae494e713df7eb61b12: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.018s
I20240503 02:16:13.067772 18751 raft_consensus.cc:348] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:13.068058 18751 raft_consensus.cc:374] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:13.068176 18751 raft_consensus.cc:729] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Initialized, Role: FOLLOWER
I20240503 02:16:13.068423 18751 consensus_queue.cc:260] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 1cd8517d70d94ae494e713df7eb61b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:13.068589 18749 tablet_bootstrap.cc:492] T 628548a443e64c6fae75070970bba761 P 4325d7d0ed484ad78a6ee54e9a003431: No bootstrap required, opened a new log
I20240503 02:16:13.068730 18749 ts_tablet_manager.cc:1367] T 628548a443e64c6fae75070970bba761 P 4325d7d0ed484ad78a6ee54e9a003431: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.005s
I20240503 02:16:13.069132 18751 ts_tablet_manager.cc:1398] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 1cd8517d70d94ae494e713df7eb61b12: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240503 02:16:13.069658 18759 raft_consensus.cc:482] T 628548a443e64c6fae75070970bba761 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:13.069880 18759 raft_consensus.cc:504] T 628548a443e64c6fae75070970bba761 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:13.070415 18750 tablet_bootstrap.cc:492] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630: No bootstrap required, opened a new log
I20240503 02:16:13.070560 18750 ts_tablet_manager.cc:1367] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.002s
I20240503 02:16:13.070986 18749 raft_consensus.cc:348] T 628548a443e64c6fae75070970bba761 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:13.071218 18749 raft_consensus.cc:374] T 628548a443e64c6fae75070970bba761 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:13.071326 18749 raft_consensus.cc:729] T 628548a443e64c6fae75070970bba761 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4325d7d0ed484ad78a6ee54e9a003431, State: Initialized, Role: FOLLOWER
I20240503 02:16:13.071249 18750 raft_consensus.cc:348] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:13.071499 18750 raft_consensus.cc:374] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:13.071604 18750 raft_consensus.cc:729] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Initialized, Role: FOLLOWER
I20240503 02:16:13.071610 18749 consensus_queue.cc:260] T 628548a443e64c6fae75070970bba761 P 4325d7d0ed484ad78a6ee54e9a003431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:13.071866 18750 consensus_queue.cc:260] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:13.072031 18759 leader_election.cc:288] T 628548a443e64c6fae75070970bba761 P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703), 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837)
I20240503 02:16:13.072487 18750 ts_tablet_manager.cc:1398] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:16:13.074965 18749 ts_tablet_manager.cc:1398] T 628548a443e64c6fae75070970bba761 P 4325d7d0ed484ad78a6ee54e9a003431: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 02:16:13.080418 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "628548a443e64c6fae75070970bba761" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431" is_pre_election: true
I20240503 02:16:13.080967 18155 raft_consensus.cc:2454] T 628548a443e64c6fae75070970bba761 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cd8517d70d94ae494e713df7eb61b12 in term 0.
I20240503 02:16:13.081669 17958 leader_election.cc:302] T 628548a443e64c6fae75070970bba761 P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cd8517d70d94ae494e713df7eb61b12, 4325d7d0ed484ad78a6ee54e9a003431; no voters:
I20240503 02:16:13.082126 18759 raft_consensus.cc:2781] T 628548a443e64c6fae75070970bba761 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:16:13.082239 18759 raft_consensus.cc:482] T 628548a443e64c6fae75070970bba761 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:16:13.082336 18759 raft_consensus.cc:3037] T 628548a443e64c6fae75070970bba761 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:13.082782 17764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "628548a443e64c6fae75070970bba761" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836b43a7ee9146948319b8d37434b630" is_pre_election: true
I20240503 02:16:13.083166 17764 raft_consensus.cc:2454] T 628548a443e64c6fae75070970bba761 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cd8517d70d94ae494e713df7eb61b12 in term 0.
I20240503 02:16:13.084615 18759 raft_consensus.cc:504] T 628548a443e64c6fae75070970bba761 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:13.085208 18759 leader_election.cc:288] T 628548a443e64c6fae75070970bba761 P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 1 election: Requested vote from peers 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703), 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837)
I20240503 02:16:13.085554 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "628548a443e64c6fae75070970bba761" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431"
I20240503 02:16:13.085685 17764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "628548a443e64c6fae75070970bba761" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836b43a7ee9146948319b8d37434b630"
I20240503 02:16:13.085767 18155 raft_consensus.cc:3037] T 628548a443e64c6fae75070970bba761 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:13.085870 17764 raft_consensus.cc:3037] T 628548a443e64c6fae75070970bba761 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:13.088862 18155 raft_consensus.cc:2454] T 628548a443e64c6fae75070970bba761 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cd8517d70d94ae494e713df7eb61b12 in term 1.
I20240503 02:16:13.088863 17764 raft_consensus.cc:2454] T 628548a443e64c6fae75070970bba761 P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cd8517d70d94ae494e713df7eb61b12 in term 1.
I20240503 02:16:13.089236 17958 leader_election.cc:302] T 628548a443e64c6fae75070970bba761 P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cd8517d70d94ae494e713df7eb61b12, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:13.089479 18759 raft_consensus.cc:2781] T 628548a443e64c6fae75070970bba761 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:16:13.089922 18759 raft_consensus.cc:686] T 628548a443e64c6fae75070970bba761 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 LEADER]: Becoming Leader. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Running, Role: LEADER
I20240503 02:16:13.090443 18759 consensus_queue.cc:237] T 628548a443e64c6fae75070970bba761 P 1cd8517d70d94ae494e713df7eb61b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:13.094575 17621 catalog_manager.cc:5484] T 628548a443e64c6fae75070970bba761 P 1cd8517d70d94ae494e713df7eb61b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3). New cstate: current_term: 1 leader_uuid: "1cd8517d70d94ae494e713df7eb61b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } }
W20240503 02:16:13.107013 18201 tablet.cc:2369] T 628548a443e64c6fae75070970bba761 P 4325d7d0ed484ad78a6ee54e9a003431: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:16:13.109099 18759 raft_consensus.cc:482] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:13.109287 18759 raft_consensus.cc:504] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:13.109956 18759 leader_election.cc:288] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647), 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837)
I20240503 02:16:13.110791 17764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5812cc63a1f4aa2b8efdcabb1bf5b29" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836b43a7ee9146948319b8d37434b630" is_pre_election: true
I20240503 02:16:13.111018 17764 raft_consensus.cc:2454] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cd8517d70d94ae494e713df7eb61b12 in term 0.
I20240503 02:16:13.111426 17958 leader_election.cc:302] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cd8517d70d94ae494e713df7eb61b12, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:13.111755 18759 raft_consensus.cc:2781] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:16:13.111858 18759 raft_consensus.cc:482] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:16:13.111989 18759 raft_consensus.cc:3037] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:13.114801 18759 raft_consensus.cc:504] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:13.115448 18759 leader_election.cc:288] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 1 election: Requested vote from peers 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647), 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837)
I20240503 02:16:13.115940 17764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5812cc63a1f4aa2b8efdcabb1bf5b29" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836b43a7ee9146948319b8d37434b630"
I20240503 02:16:13.116149 17764 raft_consensus.cc:3037] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:13.116677 17895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5812cc63a1f4aa2b8efdcabb1bf5b29" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833" is_pre_election: true
I20240503 02:16:13.116741 17894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5812cc63a1f4aa2b8efdcabb1bf5b29" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
I20240503 02:16:13.117187 17894 raft_consensus.cc:3037] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:13.118973 17764 raft_consensus.cc:2454] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cd8517d70d94ae494e713df7eb61b12 in term 1.
I20240503 02:16:13.119361 17958 leader_election.cc:302] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cd8517d70d94ae494e713df7eb61b12, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:13.119653 18759 raft_consensus.cc:2781] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:16:13.119835 18759 raft_consensus.cc:686] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 LEADER]: Becoming Leader. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Running, Role: LEADER
I20240503 02:16:13.120147 17894 raft_consensus.cc:2454] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cd8517d70d94ae494e713df7eb61b12 in term 1.
I20240503 02:16:13.120113 18759 consensus_queue.cc:237] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 1cd8517d70d94ae494e713df7eb61b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:13.122560 17621 catalog_manager.cc:5484] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 1cd8517d70d94ae494e713df7eb61b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3). New cstate: current_term: 1 leader_uuid: "1cd8517d70d94ae494e713df7eb61b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } }
W20240503 02:16:13.141758 18071 tablet.cc:2369] T 628548a443e64c6fae75070970bba761 P 1cd8517d70d94ae494e713df7eb61b12: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:16:13.183154 17941 tablet.cc:2369] T 6ccff572af764f51938873b46b6622d9 P 0f3838ee7ad04e94aa65ac750dad0833: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:16:13.213176 17810 tablet.cc:2369] T 628548a443e64c6fae75070970bba761 P 836b43a7ee9146948319b8d37434b630: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:16:13.222741 18758 raft_consensus.cc:482] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:13.223012 18758 raft_consensus.cc:504] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:13.224637 18758 leader_election.cc:288] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703), 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647)
I20240503 02:16:13.231716 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ccff572af764f51938873b46b6622d9" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431" is_pre_election: true
I20240503 02:16:13.231719 17894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ccff572af764f51938873b46b6622d9" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833" is_pre_election: true
I20240503 02:16:13.231982 17894 raft_consensus.cc:2454] T 6ccff572af764f51938873b46b6622d9 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 0.
I20240503 02:16:13.231982 18155 raft_consensus.cc:2454] T 6ccff572af764f51938873b46b6622d9 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 0.
I20240503 02:16:13.232409 17697 leader_election.cc:302] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4325d7d0ed484ad78a6ee54e9a003431, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:13.232728 18758 raft_consensus.cc:2781] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:16:13.232854 18758 raft_consensus.cc:482] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:16:13.232949 18758 raft_consensus.cc:3037] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:13.234874 18758 raft_consensus.cc:504] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:13.235412 18758 leader_election.cc:288] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 1 election: Requested vote from peers 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703), 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647)
I20240503 02:16:13.235836 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ccff572af764f51938873b46b6622d9" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431"
I20240503 02:16:13.235953 17894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ccff572af764f51938873b46b6622d9" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
I20240503 02:16:13.236059 18155 raft_consensus.cc:3037] T 6ccff572af764f51938873b46b6622d9 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:13.236181 17894 raft_consensus.cc:3037] T 6ccff572af764f51938873b46b6622d9 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:13.237939 18155 raft_consensus.cc:2454] T 6ccff572af764f51938873b46b6622d9 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 1.
I20240503 02:16:13.238284 17697 leader_election.cc:302] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4325d7d0ed484ad78a6ee54e9a003431, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:13.238523 18758 raft_consensus.cc:2781] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:16:13.238667 17894 raft_consensus.cc:2454] T 6ccff572af764f51938873b46b6622d9 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 1.
I20240503 02:16:13.239027 18758 raft_consensus.cc:686] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630 [term 1 LEADER]: Becoming Leader. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Running, Role: LEADER
I20240503 02:16:13.239501 18758 consensus_queue.cc:237] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:13.244319 17621 catalog_manager.cc:5484] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630 reported cstate change: term changed from 0 to 1, leader changed from <none> to 836b43a7ee9146948319b8d37434b630 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "836b43a7ee9146948319b8d37434b630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } }
I20240503 02:16:13.507297 18759 consensus_queue.cc:1035] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 1cd8517d70d94ae494e713df7eb61b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:16:13.515866 18759 consensus_queue.cc:1035] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 1cd8517d70d94ae494e713df7eb61b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:16:13.518709 18768 consensus_queue.cc:1035] T 628548a443e64c6fae75070970bba761 P 1cd8517d70d94ae494e713df7eb61b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:16:13.528023 18759 consensus_queue.cc:1035] T 628548a443e64c6fae75070970bba761 P 1cd8517d70d94ae494e713df7eb61b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:16:13.600600 18775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:16:13.600948 18775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:16:13.608297 18775 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 02:16:13.761397 17894 raft_consensus.cc:1261] T 6ccff572af764f51938873b46b6622d9 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Refusing update from remote peer 836b43a7ee9146948319b8d37434b630: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:16:13.762679 18155 raft_consensus.cc:1261] T 6ccff572af764f51938873b46b6622d9 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Refusing update from remote peer 836b43a7ee9146948319b8d37434b630: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:16:13.763145 18758 consensus_queue.cc:1035] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:16:13.767849 18758 consensus_queue.cc:1035] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:16:13.810864 18778 mvcc.cc:205] Tried to move back new op lower bound from 7023421742115622912 to 7023421739994005504. Current Snapshot: MvccSnapshot[applied={T|T < 7023421742115622912}]
I20240503 02:16:13.826546 18818 mvcc.cc:205] Tried to move back new op lower bound from 7023421742115622912 to 7023421739994005504. Current Snapshot: MvccSnapshot[applied={T|T < 7023421742115622912}]
INSERT report
rows total: 2000
time total: 233.195 ms
time per row: 0.116597 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 02:16:14.920331 17616 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44922:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:16:14.929543 17875 tablet_service.cc:1467] Processing CreateTablet for tablet 022740e08fd644da9d1eb22b3a8a66c0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f84a15ad000f48c2838d72859952d3e9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:14.929967 17875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 022740e08fd644da9d1eb22b3a8a66c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:14.930635 18134 tablet_service.cc:1467] Processing CreateTablet for tablet 022740e08fd644da9d1eb22b3a8a66c0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f84a15ad000f48c2838d72859952d3e9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:14.931025 18134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 022740e08fd644da9d1eb22b3a8a66c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:14.934255 18005 tablet_service.cc:1467] Processing CreateTablet for tablet 022740e08fd644da9d1eb22b3a8a66c0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f84a15ad000f48c2838d72859952d3e9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:14.934290 18003 tablet_service.cc:1467] Processing CreateTablet for tablet 9ac1c650360f4088ba5cc339cee6d45e (DEFAULT_TABLE table=rebalance_test_table_1 [id=f84a15ad000f48c2838d72859952d3e9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:14.934728 18003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ac1c650360f4088ba5cc339cee6d45e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:14.934804 18004 tablet_service.cc:1467] Processing CreateTablet for tablet 49a29f557c60494c855ba25758b6518b (DEFAULT_TABLE table=rebalance_test_table_1 [id=f84a15ad000f48c2838d72859952d3e9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:14.935174 18004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49a29f557c60494c855ba25758b6518b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:14.935344 18005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 022740e08fd644da9d1eb22b3a8a66c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:14.936676 17873 tablet_service.cc:1467] Processing CreateTablet for tablet 9ac1c650360f4088ba5cc339cee6d45e (DEFAULT_TABLE table=rebalance_test_table_1 [id=f84a15ad000f48c2838d72859952d3e9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:14.936802 17874 tablet_service.cc:1467] Processing CreateTablet for tablet 49a29f557c60494c855ba25758b6518b (DEFAULT_TABLE table=rebalance_test_table_1 [id=f84a15ad000f48c2838d72859952d3e9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:14.937150 17873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ac1c650360f4088ba5cc339cee6d45e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:14.937747 17874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49a29f557c60494c855ba25758b6518b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:14.942184 17742 tablet_service.cc:1467] Processing CreateTablet for tablet 9ac1c650360f4088ba5cc339cee6d45e (DEFAULT_TABLE table=rebalance_test_table_1 [id=f84a15ad000f48c2838d72859952d3e9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:14.942291 18135 tablet_service.cc:1467] Processing CreateTablet for tablet 49a29f557c60494c855ba25758b6518b (DEFAULT_TABLE table=rebalance_test_table_1 [id=f84a15ad000f48c2838d72859952d3e9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:14.942600 17742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ac1c650360f4088ba5cc339cee6d45e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:14.942811 18135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49a29f557c60494c855ba25758b6518b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:14.950179 18845 tablet_bootstrap.cc:492] T 022740e08fd644da9d1eb22b3a8a66c0 P 1cd8517d70d94ae494e713df7eb61b12: Bootstrap starting.
I20240503 02:16:14.952239 18845 tablet_bootstrap.cc:654] T 022740e08fd644da9d1eb22b3a8a66c0 P 1cd8517d70d94ae494e713df7eb61b12: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:14.957582 18843 tablet_bootstrap.cc:492] T 022740e08fd644da9d1eb22b3a8a66c0 P 0f3838ee7ad04e94aa65ac750dad0833: Bootstrap starting.
I20240503 02:16:14.959707 18843 tablet_bootstrap.cc:654] T 022740e08fd644da9d1eb22b3a8a66c0 P 0f3838ee7ad04e94aa65ac750dad0833: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:14.961934 18845 tablet_bootstrap.cc:492] T 022740e08fd644da9d1eb22b3a8a66c0 P 1cd8517d70d94ae494e713df7eb61b12: No bootstrap required, opened a new log
I20240503 02:16:14.962221 18845 ts_tablet_manager.cc:1367] T 022740e08fd644da9d1eb22b3a8a66c0 P 1cd8517d70d94ae494e713df7eb61b12: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.001s
I20240503 02:16:14.963315 18845 raft_consensus.cc:348] T 022740e08fd644da9d1eb22b3a8a66c0 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:14.963747 18845 raft_consensus.cc:374] T 022740e08fd644da9d1eb22b3a8a66c0 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:14.963923 18845 raft_consensus.cc:729] T 022740e08fd644da9d1eb22b3a8a66c0 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Initialized, Role: FOLLOWER
I20240503 02:16:14.964350 18845 consensus_queue.cc:260] T 022740e08fd644da9d1eb22b3a8a66c0 P 1cd8517d70d94ae494e713df7eb61b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:14.965157 18845 ts_tablet_manager.cc:1398] T 022740e08fd644da9d1eb22b3a8a66c0 P 1cd8517d70d94ae494e713df7eb61b12: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:16:14.965482 18845 tablet_bootstrap.cc:492] T 9ac1c650360f4088ba5cc339cee6d45e P 1cd8517d70d94ae494e713df7eb61b12: Bootstrap starting.
I20240503 02:16:14.967149 18845 tablet_bootstrap.cc:654] T 9ac1c650360f4088ba5cc339cee6d45e P 1cd8517d70d94ae494e713df7eb61b12: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:14.971890 18843 tablet_bootstrap.cc:492] T 022740e08fd644da9d1eb22b3a8a66c0 P 0f3838ee7ad04e94aa65ac750dad0833: No bootstrap required, opened a new log
I20240503 02:16:14.972141 18843 ts_tablet_manager.cc:1367] T 022740e08fd644da9d1eb22b3a8a66c0 P 0f3838ee7ad04e94aa65ac750dad0833: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240503 02:16:14.973776 18846 tablet_bootstrap.cc:492] T 9ac1c650360f4088ba5cc339cee6d45e P 836b43a7ee9146948319b8d37434b630: Bootstrap starting.
I20240503 02:16:14.973843 18843 raft_consensus.cc:348] T 022740e08fd644da9d1eb22b3a8a66c0 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:14.974169 18843 raft_consensus.cc:374] T 022740e08fd644da9d1eb22b3a8a66c0 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:14.974290 18843 raft_consensus.cc:729] T 022740e08fd644da9d1eb22b3a8a66c0 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Initialized, Role: FOLLOWER
I20240503 02:16:14.974521 18843 consensus_queue.cc:260] T 022740e08fd644da9d1eb22b3a8a66c0 P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:14.976161 18845 tablet_bootstrap.cc:492] T 9ac1c650360f4088ba5cc339cee6d45e P 1cd8517d70d94ae494e713df7eb61b12: No bootstrap required, opened a new log
I20240503 02:16:14.976349 18845 ts_tablet_manager.cc:1367] T 9ac1c650360f4088ba5cc339cee6d45e P 1cd8517d70d94ae494e713df7eb61b12: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20240503 02:16:14.976445 18846 tablet_bootstrap.cc:654] T 9ac1c650360f4088ba5cc339cee6d45e P 836b43a7ee9146948319b8d37434b630: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:14.977159 18845 raft_consensus.cc:348] T 9ac1c650360f4088ba5cc339cee6d45e P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:14.977433 18845 raft_consensus.cc:374] T 9ac1c650360f4088ba5cc339cee6d45e P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:14.977572 18845 raft_consensus.cc:729] T 9ac1c650360f4088ba5cc339cee6d45e P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Initialized, Role: FOLLOWER
I20240503 02:16:14.977901 18845 consensus_queue.cc:260] T 9ac1c650360f4088ba5cc339cee6d45e P 1cd8517d70d94ae494e713df7eb61b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:14.978374 18844 tablet_bootstrap.cc:492] T 022740e08fd644da9d1eb22b3a8a66c0 P 4325d7d0ed484ad78a6ee54e9a003431: Bootstrap starting.
I20240503 02:16:14.978612 18843 ts_tablet_manager.cc:1398] T 022740e08fd644da9d1eb22b3a8a66c0 P 0f3838ee7ad04e94aa65ac750dad0833: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:16:14.978863 18845 ts_tablet_manager.cc:1398] T 9ac1c650360f4088ba5cc339cee6d45e P 1cd8517d70d94ae494e713df7eb61b12: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:16:14.978971 18843 tablet_bootstrap.cc:492] T 9ac1c650360f4088ba5cc339cee6d45e P 0f3838ee7ad04e94aa65ac750dad0833: Bootstrap starting.
I20240503 02:16:14.980151 18844 tablet_bootstrap.cc:654] T 022740e08fd644da9d1eb22b3a8a66c0 P 4325d7d0ed484ad78a6ee54e9a003431: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:14.980664 18843 tablet_bootstrap.cc:654] T 9ac1c650360f4088ba5cc339cee6d45e P 0f3838ee7ad04e94aa65ac750dad0833: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:14.982050 18846 tablet_bootstrap.cc:492] T 9ac1c650360f4088ba5cc339cee6d45e P 836b43a7ee9146948319b8d37434b630: No bootstrap required, opened a new log
I20240503 02:16:14.982306 18846 ts_tablet_manager.cc:1367] T 9ac1c650360f4088ba5cc339cee6d45e P 836b43a7ee9146948319b8d37434b630: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s
I20240503 02:16:14.982764 18845 tablet_bootstrap.cc:492] T 49a29f557c60494c855ba25758b6518b P 1cd8517d70d94ae494e713df7eb61b12: Bootstrap starting.
I20240503 02:16:14.983181 18846 raft_consensus.cc:348] T 9ac1c650360f4088ba5cc339cee6d45e P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:14.983538 18846 raft_consensus.cc:374] T 9ac1c650360f4088ba5cc339cee6d45e P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:14.983680 18846 raft_consensus.cc:729] T 9ac1c650360f4088ba5cc339cee6d45e P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Initialized, Role: FOLLOWER
I20240503 02:16:14.983989 18846 consensus_queue.cc:260] T 9ac1c650360f4088ba5cc339cee6d45e P 836b43a7ee9146948319b8d37434b630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:14.984500 18845 tablet_bootstrap.cc:654] T 49a29f557c60494c855ba25758b6518b P 1cd8517d70d94ae494e713df7eb61b12: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:14.986536 18846 ts_tablet_manager.cc:1398] T 9ac1c650360f4088ba5cc339cee6d45e P 836b43a7ee9146948319b8d37434b630: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:16:14.986840 18844 tablet_bootstrap.cc:492] T 022740e08fd644da9d1eb22b3a8a66c0 P 4325d7d0ed484ad78a6ee54e9a003431: No bootstrap required, opened a new log
I20240503 02:16:14.987046 18844 ts_tablet_manager.cc:1367] T 022740e08fd644da9d1eb22b3a8a66c0 P 4325d7d0ed484ad78a6ee54e9a003431: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20240503 02:16:14.987568 18843 tablet_bootstrap.cc:492] T 9ac1c650360f4088ba5cc339cee6d45e P 0f3838ee7ad04e94aa65ac750dad0833: No bootstrap required, opened a new log
I20240503 02:16:14.987771 18843 ts_tablet_manager.cc:1367] T 9ac1c650360f4088ba5cc339cee6d45e P 0f3838ee7ad04e94aa65ac750dad0833: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20240503 02:16:14.987964 18844 raft_consensus.cc:348] T 022740e08fd644da9d1eb22b3a8a66c0 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:14.988241 18844 raft_consensus.cc:374] T 022740e08fd644da9d1eb22b3a8a66c0 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:14.988371 18844 raft_consensus.cc:729] T 022740e08fd644da9d1eb22b3a8a66c0 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4325d7d0ed484ad78a6ee54e9a003431, State: Initialized, Role: FOLLOWER
I20240503 02:16:14.988575 18843 raft_consensus.cc:348] T 9ac1c650360f4088ba5cc339cee6d45e P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:14.988838 18843 raft_consensus.cc:374] T 9ac1c650360f4088ba5cc339cee6d45e P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:14.988677 18844 consensus_queue.cc:260] T 022740e08fd644da9d1eb22b3a8a66c0 P 4325d7d0ed484ad78a6ee54e9a003431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:14.988955 18843 raft_consensus.cc:729] T 9ac1c650360f4088ba5cc339cee6d45e P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Initialized, Role: FOLLOWER
I20240503 02:16:14.989245 18843 consensus_queue.cc:260] T 9ac1c650360f4088ba5cc339cee6d45e P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:14.989807 18843 ts_tablet_manager.cc:1398] T 9ac1c650360f4088ba5cc339cee6d45e P 0f3838ee7ad04e94aa65ac750dad0833: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:16:14.989935 18845 tablet_bootstrap.cc:492] T 49a29f557c60494c855ba25758b6518b P 1cd8517d70d94ae494e713df7eb61b12: No bootstrap required, opened a new log
I20240503 02:16:14.990069 18845 ts_tablet_manager.cc:1367] T 49a29f557c60494c855ba25758b6518b P 1cd8517d70d94ae494e713df7eb61b12: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240503 02:16:14.990231 18843 tablet_bootstrap.cc:492] T 49a29f557c60494c855ba25758b6518b P 0f3838ee7ad04e94aa65ac750dad0833: Bootstrap starting.
I20240503 02:16:14.990813 18845 raft_consensus.cc:348] T 49a29f557c60494c855ba25758b6518b P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:14.991092 18845 raft_consensus.cc:374] T 49a29f557c60494c855ba25758b6518b P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:14.991253 18845 raft_consensus.cc:729] T 49a29f557c60494c855ba25758b6518b P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Initialized, Role: FOLLOWER
I20240503 02:16:14.991546 18845 consensus_queue.cc:260] T 49a29f557c60494c855ba25758b6518b P 1cd8517d70d94ae494e713df7eb61b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:14.992359 18843 tablet_bootstrap.cc:654] T 49a29f557c60494c855ba25758b6518b P 0f3838ee7ad04e94aa65ac750dad0833: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:14.994210 18844 ts_tablet_manager.cc:1398] T 022740e08fd644da9d1eb22b3a8a66c0 P 4325d7d0ed484ad78a6ee54e9a003431: Time spent starting tablet: real 0.007s user 0.000s sys 0.005s
I20240503 02:16:14.994632 18843 tablet_bootstrap.cc:492] T 49a29f557c60494c855ba25758b6518b P 0f3838ee7ad04e94aa65ac750dad0833: No bootstrap required, opened a new log
I20240503 02:16:14.994755 18843 ts_tablet_manager.cc:1367] T 49a29f557c60494c855ba25758b6518b P 0f3838ee7ad04e94aa65ac750dad0833: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240503 02:16:14.994798 18845 ts_tablet_manager.cc:1398] T 49a29f557c60494c855ba25758b6518b P 1cd8517d70d94ae494e713df7eb61b12: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 02:16:14.994781 18844 tablet_bootstrap.cc:492] T 49a29f557c60494c855ba25758b6518b P 4325d7d0ed484ad78a6ee54e9a003431: Bootstrap starting.
I20240503 02:16:14.995543 18843 raft_consensus.cc:348] T 49a29f557c60494c855ba25758b6518b P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:14.995760 18843 raft_consensus.cc:374] T 49a29f557c60494c855ba25758b6518b P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:14.995847 18843 raft_consensus.cc:729] T 49a29f557c60494c855ba25758b6518b P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Initialized, Role: FOLLOWER
I20240503 02:16:14.996253 18843 consensus_queue.cc:260] T 49a29f557c60494c855ba25758b6518b P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:14.996529 18844 tablet_bootstrap.cc:654] T 49a29f557c60494c855ba25758b6518b P 4325d7d0ed484ad78a6ee54e9a003431: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:14.996750 18838 raft_consensus.cc:482] T 9ac1c650360f4088ba5cc339cee6d45e P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:14.996892 18838 raft_consensus.cc:504] T 9ac1c650360f4088ba5cc339cee6d45e P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:14.997313 18843 ts_tablet_manager.cc:1398] T 49a29f557c60494c855ba25758b6518b P 0f3838ee7ad04e94aa65ac750dad0833: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:16:14.997604 18838 leader_election.cc:288] T 9ac1c650360f4088ba5cc339cee6d45e P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647), 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837)
I20240503 02:16:14.998323 17895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac1c650360f4088ba5cc339cee6d45e" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833" is_pre_election: true
I20240503 02:16:14.998533 17895 raft_consensus.cc:2454] T 9ac1c650360f4088ba5cc339cee6d45e P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cd8517d70d94ae494e713df7eb61b12 in term 0.
I20240503 02:16:14.998553 17763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac1c650360f4088ba5cc339cee6d45e" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836b43a7ee9146948319b8d37434b630" is_pre_election: true
I20240503 02:16:14.998740 17763 raft_consensus.cc:2454] T 9ac1c650360f4088ba5cc339cee6d45e P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cd8517d70d94ae494e713df7eb61b12 in term 0.
I20240503 02:16:14.999500 17958 leader_election.cc:302] T 9ac1c650360f4088ba5cc339cee6d45e P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cd8517d70d94ae494e713df7eb61b12, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:15.001019 18838 raft_consensus.cc:2781] T 9ac1c650360f4088ba5cc339cee6d45e P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:16:15.001150 18838 raft_consensus.cc:482] T 9ac1c650360f4088ba5cc339cee6d45e P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:16:15.001231 18838 raft_consensus.cc:3037] T 9ac1c650360f4088ba5cc339cee6d45e P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:15.003502 18838 raft_consensus.cc:504] T 9ac1c650360f4088ba5cc339cee6d45e P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:15.003927 18838 leader_election.cc:288] T 9ac1c650360f4088ba5cc339cee6d45e P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 1 election: Requested vote from peers 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647), 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837)
I20240503 02:16:15.004523 18844 tablet_bootstrap.cc:492] T 49a29f557c60494c855ba25758b6518b P 4325d7d0ed484ad78a6ee54e9a003431: No bootstrap required, opened a new log
I20240503 02:16:15.004649 17895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac1c650360f4088ba5cc339cee6d45e" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
I20240503 02:16:15.004746 18844 ts_tablet_manager.cc:1367] T 49a29f557c60494c855ba25758b6518b P 4325d7d0ed484ad78a6ee54e9a003431: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20240503 02:16:15.004730 17763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac1c650360f4088ba5cc339cee6d45e" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836b43a7ee9146948319b8d37434b630"
I20240503 02:16:15.004837 17895 raft_consensus.cc:3037] T 9ac1c650360f4088ba5cc339cee6d45e P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:15.004951 17763 raft_consensus.cc:3037] T 9ac1c650360f4088ba5cc339cee6d45e P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:15.005612 18844 raft_consensus.cc:348] T 49a29f557c60494c855ba25758b6518b P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:15.005869 18844 raft_consensus.cc:374] T 49a29f557c60494c855ba25758b6518b P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:15.005949 18844 raft_consensus.cc:729] T 49a29f557c60494c855ba25758b6518b P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4325d7d0ed484ad78a6ee54e9a003431, State: Initialized, Role: FOLLOWER
I20240503 02:16:15.006198 18844 consensus_queue.cc:260] T 49a29f557c60494c855ba25758b6518b P 4325d7d0ed484ad78a6ee54e9a003431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:15.006820 18844 ts_tablet_manager.cc:1398] T 49a29f557c60494c855ba25758b6518b P 4325d7d0ed484ad78a6ee54e9a003431: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:16:15.007089 17895 raft_consensus.cc:2454] T 9ac1c650360f4088ba5cc339cee6d45e P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cd8517d70d94ae494e713df7eb61b12 in term 1.
I20240503 02:16:15.007449 17958 leader_election.cc:302] T 9ac1c650360f4088ba5cc339cee6d45e P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 1cd8517d70d94ae494e713df7eb61b12; no voters:
I20240503 02:16:15.007691 18858 raft_consensus.cc:2781] T 9ac1c650360f4088ba5cc339cee6d45e P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:16:15.007891 17763 raft_consensus.cc:2454] T 9ac1c650360f4088ba5cc339cee6d45e P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cd8517d70d94ae494e713df7eb61b12 in term 1.
I20240503 02:16:15.007917 18858 raft_consensus.cc:686] T 9ac1c650360f4088ba5cc339cee6d45e P 1cd8517d70d94ae494e713df7eb61b12 [term 1 LEADER]: Becoming Leader. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Running, Role: LEADER
I20240503 02:16:15.008183 18858 consensus_queue.cc:237] T 9ac1c650360f4088ba5cc339cee6d45e P 1cd8517d70d94ae494e713df7eb61b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:15.010668 17616 catalog_manager.cc:5484] T 9ac1c650360f4088ba5cc339cee6d45e P 1cd8517d70d94ae494e713df7eb61b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3). New cstate: current_term: 1 leader_uuid: "1cd8517d70d94ae494e713df7eb61b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } }
I20240503 02:16:15.041435 18850 raft_consensus.cc:482] T 022740e08fd644da9d1eb22b3a8a66c0 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:15.041630 18850 raft_consensus.cc:504] T 022740e08fd644da9d1eb22b3a8a66c0 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:15.042810 18855 raft_consensus.cc:482] T 022740e08fd644da9d1eb22b3a8a66c0 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:15.042997 18855 raft_consensus.cc:504] T 022740e08fd644da9d1eb22b3a8a66c0 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:15.043282 18850 leader_election.cc:288] T 022740e08fd644da9d1eb22b3a8a66c0 P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703), 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759)
I20240503 02:16:15.044795 18855 leader_election.cc:288] T 022740e08fd644da9d1eb22b3a8a66c0 P 4325d7d0ed484ad78a6ee54e9a003431 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647), 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759)
I20240503 02:16:15.057170 17895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "022740e08fd644da9d1eb22b3a8a66c0" candidate_uuid: "4325d7d0ed484ad78a6ee54e9a003431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833" is_pre_election: true
I20240503 02:16:15.057525 17895 raft_consensus.cc:2454] T 022740e08fd644da9d1eb22b3a8a66c0 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4325d7d0ed484ad78a6ee54e9a003431 in term 0.
I20240503 02:16:15.058269 18088 leader_election.cc:302] T 022740e08fd644da9d1eb22b3a8a66c0 P 4325d7d0ed484ad78a6ee54e9a003431 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 4325d7d0ed484ad78a6ee54e9a003431; no voters:
I20240503 02:16:15.058753 18855 raft_consensus.cc:2781] T 022740e08fd644da9d1eb22b3a8a66c0 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:16:15.058892 18855 raft_consensus.cc:482] T 022740e08fd644da9d1eb22b3a8a66c0 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:16:15.058979 18855 raft_consensus.cc:3037] T 022740e08fd644da9d1eb22b3a8a66c0 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:15.062300 18855 raft_consensus.cc:504] T 022740e08fd644da9d1eb22b3a8a66c0 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:15.063581 17895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "022740e08fd644da9d1eb22b3a8a66c0" candidate_uuid: "4325d7d0ed484ad78a6ee54e9a003431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
I20240503 02:16:15.063787 17895 raft_consensus.cc:3037] T 022740e08fd644da9d1eb22b3a8a66c0 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:15.065219 18025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "022740e08fd644da9d1eb22b3a8a66c0" candidate_uuid: "4325d7d0ed484ad78a6ee54e9a003431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd8517d70d94ae494e713df7eb61b12" is_pre_election: true
I20240503 02:16:15.065622 18855 leader_election.cc:288] T 022740e08fd644da9d1eb22b3a8a66c0 P 4325d7d0ed484ad78a6ee54e9a003431 [CANDIDATE]: Term 1 election: Requested vote from peers 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647), 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759)
I20240503 02:16:15.065639 18025 raft_consensus.cc:2454] T 022740e08fd644da9d1eb22b3a8a66c0 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4325d7d0ed484ad78a6ee54e9a003431 in term 0.
I20240503 02:16:15.065725 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "022740e08fd644da9d1eb22b3a8a66c0" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431" is_pre_election: true
I20240503 02:16:15.066017 18155 raft_consensus.cc:2379] T 022740e08fd644da9d1eb22b3a8a66c0 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0f3838ee7ad04e94aa65ac750dad0833 in current term 1: Already voted for candidate 4325d7d0ed484ad78a6ee54e9a003431 in this term.
I20240503 02:16:15.066298 18025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "022740e08fd644da9d1eb22b3a8a66c0" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd8517d70d94ae494e713df7eb61b12" is_pre_election: true
I20240503 02:16:15.066478 18025 raft_consensus.cc:2454] T 022740e08fd644da9d1eb22b3a8a66c0 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 0.
I20240503 02:16:15.066574 18024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "022740e08fd644da9d1eb22b3a8a66c0" candidate_uuid: "4325d7d0ed484ad78a6ee54e9a003431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:15.066840 18024 raft_consensus.cc:3037] T 022740e08fd644da9d1eb22b3a8a66c0 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:15.066952 17895 raft_consensus.cc:2454] T 022740e08fd644da9d1eb22b3a8a66c0 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4325d7d0ed484ad78a6ee54e9a003431 in term 1.
I20240503 02:16:15.067015 17831 leader_election.cc:302] T 022740e08fd644da9d1eb22b3a8a66c0 P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 1cd8517d70d94ae494e713df7eb61b12; no voters: 4325d7d0ed484ad78a6ee54e9a003431
I20240503 02:16:15.067335 18088 leader_election.cc:302] T 022740e08fd644da9d1eb22b3a8a66c0 P 4325d7d0ed484ad78a6ee54e9a003431 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 4325d7d0ed484ad78a6ee54e9a003431; no voters:
I20240503 02:16:15.067467 18850 raft_consensus.cc:2741] T 022740e08fd644da9d1eb22b3a8a66c0 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240503 02:16:15.067708 18855 raft_consensus.cc:2781] T 022740e08fd644da9d1eb22b3a8a66c0 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:16:15.068192 18855 raft_consensus.cc:686] T 022740e08fd644da9d1eb22b3a8a66c0 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 LEADER]: Becoming Leader. State: Replica: 4325d7d0ed484ad78a6ee54e9a003431, State: Running, Role: LEADER
I20240503 02:16:15.068693 18855 consensus_queue.cc:237] T 022740e08fd644da9d1eb22b3a8a66c0 P 4325d7d0ed484ad78a6ee54e9a003431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:15.069384 18024 raft_consensus.cc:2454] T 022740e08fd644da9d1eb22b3a8a66c0 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4325d7d0ed484ad78a6ee54e9a003431 in term 1.
I20240503 02:16:15.072175 17616 catalog_manager.cc:5484] T 022740e08fd644da9d1eb22b3a8a66c0 P 4325d7d0ed484ad78a6ee54e9a003431 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4). New cstate: current_term: 1 leader_uuid: "4325d7d0ed484ad78a6ee54e9a003431" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } }
I20240503 02:16:15.213414 18850 raft_consensus.cc:482] T 49a29f557c60494c855ba25758b6518b P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:15.213629 18850 raft_consensus.cc:504] T 49a29f557c60494c855ba25758b6518b P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:15.214327 18850 leader_election.cc:288] T 49a29f557c60494c855ba25758b6518b P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703), 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759)
I20240503 02:16:15.214856 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a29f557c60494c855ba25758b6518b" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431" is_pre_election: true
I20240503 02:16:15.214866 18024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a29f557c60494c855ba25758b6518b" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd8517d70d94ae494e713df7eb61b12" is_pre_election: true
I20240503 02:16:15.215152 18155 raft_consensus.cc:2454] T 49a29f557c60494c855ba25758b6518b P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 0.
I20240503 02:16:15.215147 18024 raft_consensus.cc:2454] T 49a29f557c60494c855ba25758b6518b P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 0.
I20240503 02:16:15.215580 17831 leader_election.cc:302] T 49a29f557c60494c855ba25758b6518b P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 1cd8517d70d94ae494e713df7eb61b12; no voters:
I20240503 02:16:15.215874 18850 raft_consensus.cc:2781] T 49a29f557c60494c855ba25758b6518b P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:16:15.215971 18850 raft_consensus.cc:482] T 49a29f557c60494c855ba25758b6518b P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:16:15.216037 18850 raft_consensus.cc:3037] T 49a29f557c60494c855ba25758b6518b P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:15.218088 18850 raft_consensus.cc:504] T 49a29f557c60494c855ba25758b6518b P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:15.218708 18850 leader_election.cc:288] T 49a29f557c60494c855ba25758b6518b P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 1 election: Requested vote from peers 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703), 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759)
I20240503 02:16:15.219194 18024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a29f557c60494c855ba25758b6518b" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:15.219167 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a29f557c60494c855ba25758b6518b" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431"
I20240503 02:16:15.219426 18024 raft_consensus.cc:3037] T 49a29f557c60494c855ba25758b6518b P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:15.219420 18155 raft_consensus.cc:3037] T 49a29f557c60494c855ba25758b6518b P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:15.222164 18024 raft_consensus.cc:2454] T 49a29f557c60494c855ba25758b6518b P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 1.
I20240503 02:16:15.222164 18155 raft_consensus.cc:2454] T 49a29f557c60494c855ba25758b6518b P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 1.
I20240503 02:16:15.222721 17828 leader_election.cc:302] T 49a29f557c60494c855ba25758b6518b P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 1cd8517d70d94ae494e713df7eb61b12, 4325d7d0ed484ad78a6ee54e9a003431; no voters:
I20240503 02:16:15.223021 18850 raft_consensus.cc:2781] T 49a29f557c60494c855ba25758b6518b P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:16:15.223484 18850 raft_consensus.cc:686] T 49a29f557c60494c855ba25758b6518b P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 LEADER]: Becoming Leader. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:15.223925 18850 consensus_queue.cc:237] T 49a29f557c60494c855ba25758b6518b P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:15.227228 17616 catalog_manager.cc:5484] T 49a29f557c60494c855ba25758b6518b P 0f3838ee7ad04e94aa65ac750dad0833 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2). New cstate: current_term: 1 leader_uuid: "0f3838ee7ad04e94aa65ac750dad0833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } }
I20240503 02:16:15.448027 18855 consensus_queue.cc:1035] T 022740e08fd644da9d1eb22b3a8a66c0 P 4325d7d0ed484ad78a6ee54e9a003431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:16:15.453953 18855 consensus_queue.cc:1035] T 022740e08fd644da9d1eb22b3a8a66c0 P 4325d7d0ed484ad78a6ee54e9a003431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:16:15.521262 18858 consensus_queue.cc:1035] T 9ac1c650360f4088ba5cc339cee6d45e P 1cd8517d70d94ae494e713df7eb61b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:16:15.527763 18858 consensus_queue.cc:1035] T 9ac1c650360f4088ba5cc339cee6d45e P 1cd8517d70d94ae494e713df7eb61b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:16:15.583331 18872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:16:15.583647 18872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:16:15.590943 18872 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 02:16:15.648813 18850 consensus_queue.cc:1035] T 49a29f557c60494c855ba25758b6518b P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:16:15.669195 18870 consensus_queue.cc:1035] T 49a29f557c60494c855ba25758b6518b P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 265.849 ms
time per row: 0.132924 ms
I20240503 02:16:16.002460 17619 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44922:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:16:16.011133 17742 tablet_service.cc:1467] Processing CreateTablet for tablet a8cc6cfa2b8240c9981a692e0896ef50 (DEFAULT_TABLE table=rebalance_test_table_2 [id=dd2e0b5526e04301a682be719e7e4d4d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:16.011313 17874 tablet_service.cc:1467] Processing CreateTablet for tablet a8cc6cfa2b8240c9981a692e0896ef50 (DEFAULT_TABLE table=rebalance_test_table_2 [id=dd2e0b5526e04301a682be719e7e4d4d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:16.011615 17742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8cc6cfa2b8240c9981a692e0896ef50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:16.011682 17874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8cc6cfa2b8240c9981a692e0896ef50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:16.012356 18134 tablet_service.cc:1467] Processing CreateTablet for tablet a8cc6cfa2b8240c9981a692e0896ef50 (DEFAULT_TABLE table=rebalance_test_table_2 [id=dd2e0b5526e04301a682be719e7e4d4d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:16.012780 18134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8cc6cfa2b8240c9981a692e0896ef50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:16.015059 18135 tablet_service.cc:1467] Processing CreateTablet for tablet e53bab9a159147d79a1f96269f1c1201 (DEFAULT_TABLE table=rebalance_test_table_2 [id=dd2e0b5526e04301a682be719e7e4d4d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:16.015460 18135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e53bab9a159147d79a1f96269f1c1201. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:16.015746 18133 tablet_service.cc:1467] Processing CreateTablet for tablet d3d114f397e44d07a729f6eb09f279cc (DEFAULT_TABLE table=rebalance_test_table_2 [id=dd2e0b5526e04301a682be719e7e4d4d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:16.016250 18133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3d114f397e44d07a729f6eb09f279cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:16.017673 17875 tablet_service.cc:1467] Processing CreateTablet for tablet d3d114f397e44d07a729f6eb09f279cc (DEFAULT_TABLE table=rebalance_test_table_2 [id=dd2e0b5526e04301a682be719e7e4d4d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:16.018136 17875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3d114f397e44d07a729f6eb09f279cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:16.021638 18004 tablet_service.cc:1467] Processing CreateTablet for tablet e53bab9a159147d79a1f96269f1c1201 (DEFAULT_TABLE table=rebalance_test_table_2 [id=dd2e0b5526e04301a682be719e7e4d4d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:16.022228 18004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e53bab9a159147d79a1f96269f1c1201. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:16.026206 17744 tablet_service.cc:1467] Processing CreateTablet for tablet e53bab9a159147d79a1f96269f1c1201 (DEFAULT_TABLE table=rebalance_test_table_2 [id=dd2e0b5526e04301a682be719e7e4d4d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:16.026362 18943 tablet_bootstrap.cc:492] T a8cc6cfa2b8240c9981a692e0896ef50 P 4325d7d0ed484ad78a6ee54e9a003431: Bootstrap starting.
I20240503 02:16:16.026657 17744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e53bab9a159147d79a1f96269f1c1201. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:16.027148 17743 tablet_service.cc:1467] Processing CreateTablet for tablet d3d114f397e44d07a729f6eb09f279cc (DEFAULT_TABLE table=rebalance_test_table_2 [id=dd2e0b5526e04301a682be719e7e4d4d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:16.027527 17743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3d114f397e44d07a729f6eb09f279cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:16.028229 18943 tablet_bootstrap.cc:654] T a8cc6cfa2b8240c9981a692e0896ef50 P 4325d7d0ed484ad78a6ee54e9a003431: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:16.034910 18944 tablet_bootstrap.cc:492] T d3d114f397e44d07a729f6eb09f279cc P 0f3838ee7ad04e94aa65ac750dad0833: Bootstrap starting.
I20240503 02:16:16.038528 18944 tablet_bootstrap.cc:654] T d3d114f397e44d07a729f6eb09f279cc P 0f3838ee7ad04e94aa65ac750dad0833: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:16.038796 18942 tablet_bootstrap.cc:492] T a8cc6cfa2b8240c9981a692e0896ef50 P 836b43a7ee9146948319b8d37434b630: Bootstrap starting.
I20240503 02:16:16.039721 18943 tablet_bootstrap.cc:492] T a8cc6cfa2b8240c9981a692e0896ef50 P 4325d7d0ed484ad78a6ee54e9a003431: No bootstrap required, opened a new log
I20240503 02:16:16.039916 18943 ts_tablet_manager.cc:1367] T a8cc6cfa2b8240c9981a692e0896ef50 P 4325d7d0ed484ad78a6ee54e9a003431: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.000s
I20240503 02:16:16.040596 18942 tablet_bootstrap.cc:654] T a8cc6cfa2b8240c9981a692e0896ef50 P 836b43a7ee9146948319b8d37434b630: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:16.040831 18943 raft_consensus.cc:348] T a8cc6cfa2b8240c9981a692e0896ef50 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:16.041209 18943 raft_consensus.cc:374] T a8cc6cfa2b8240c9981a692e0896ef50 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:16.041359 18943 raft_consensus.cc:729] T a8cc6cfa2b8240c9981a692e0896ef50 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4325d7d0ed484ad78a6ee54e9a003431, State: Initialized, Role: FOLLOWER
I20240503 02:16:16.041786 18943 consensus_queue.cc:260] T a8cc6cfa2b8240c9981a692e0896ef50 P 4325d7d0ed484ad78a6ee54e9a003431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:16.042903 18943 ts_tablet_manager.cc:1398] T a8cc6cfa2b8240c9981a692e0896ef50 P 4325d7d0ed484ad78a6ee54e9a003431: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:16:16.043521 18943 tablet_bootstrap.cc:492] T d3d114f397e44d07a729f6eb09f279cc P 4325d7d0ed484ad78a6ee54e9a003431: Bootstrap starting.
I20240503 02:16:16.045733 18943 tablet_bootstrap.cc:654] T d3d114f397e44d07a729f6eb09f279cc P 4325d7d0ed484ad78a6ee54e9a003431: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:16.049176 18942 tablet_bootstrap.cc:492] T a8cc6cfa2b8240c9981a692e0896ef50 P 836b43a7ee9146948319b8d37434b630: No bootstrap required, opened a new log
I20240503 02:16:16.049454 18942 ts_tablet_manager.cc:1367] T a8cc6cfa2b8240c9981a692e0896ef50 P 836b43a7ee9146948319b8d37434b630: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20240503 02:16:16.049823 18944 tablet_bootstrap.cc:492] T d3d114f397e44d07a729f6eb09f279cc P 0f3838ee7ad04e94aa65ac750dad0833: No bootstrap required, opened a new log
I20240503 02:16:16.049999 18946 tablet_bootstrap.cc:492] T e53bab9a159147d79a1f96269f1c1201 P 1cd8517d70d94ae494e713df7eb61b12: Bootstrap starting.
I20240503 02:16:16.050069 18944 ts_tablet_manager.cc:1367] T d3d114f397e44d07a729f6eb09f279cc P 0f3838ee7ad04e94aa65ac750dad0833: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20240503 02:16:16.051098 18944 raft_consensus.cc:348] T d3d114f397e44d07a729f6eb09f279cc P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:16.051390 18944 raft_consensus.cc:374] T d3d114f397e44d07a729f6eb09f279cc P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:16.051517 18944 raft_consensus.cc:729] T d3d114f397e44d07a729f6eb09f279cc P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Initialized, Role: FOLLOWER
I20240503 02:16:16.051339 18942 raft_consensus.cc:348] T a8cc6cfa2b8240c9981a692e0896ef50 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:16.051765 18942 raft_consensus.cc:374] T a8cc6cfa2b8240c9981a692e0896ef50 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:16.051931 18942 raft_consensus.cc:729] T a8cc6cfa2b8240c9981a692e0896ef50 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Initialized, Role: FOLLOWER
I20240503 02:16:16.051834 18944 consensus_queue.cc:260] T d3d114f397e44d07a729f6eb09f279cc P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:16.052105 18946 tablet_bootstrap.cc:654] T e53bab9a159147d79a1f96269f1c1201 P 1cd8517d70d94ae494e713df7eb61b12: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:16.052544 18944 ts_tablet_manager.cc:1398] T d3d114f397e44d07a729f6eb09f279cc P 0f3838ee7ad04e94aa65ac750dad0833: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:16:16.052379 18942 consensus_queue.cc:260] T a8cc6cfa2b8240c9981a692e0896ef50 P 836b43a7ee9146948319b8d37434b630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:16.053141 18942 ts_tablet_manager.cc:1398] T a8cc6cfa2b8240c9981a692e0896ef50 P 836b43a7ee9146948319b8d37434b630: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:16:16.053529 18944 tablet_bootstrap.cc:492] T a8cc6cfa2b8240c9981a692e0896ef50 P 0f3838ee7ad04e94aa65ac750dad0833: Bootstrap starting.
I20240503 02:16:16.055042 18942 tablet_bootstrap.cc:492] T e53bab9a159147d79a1f96269f1c1201 P 836b43a7ee9146948319b8d37434b630: Bootstrap starting.
I20240503 02:16:16.055219 18944 tablet_bootstrap.cc:654] T a8cc6cfa2b8240c9981a692e0896ef50 P 0f3838ee7ad04e94aa65ac750dad0833: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:16.057729 18942 tablet_bootstrap.cc:654] T e53bab9a159147d79a1f96269f1c1201 P 836b43a7ee9146948319b8d37434b630: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:16.058146 18943 tablet_bootstrap.cc:492] T d3d114f397e44d07a729f6eb09f279cc P 4325d7d0ed484ad78a6ee54e9a003431: No bootstrap required, opened a new log
I20240503 02:16:16.058362 18943 ts_tablet_manager.cc:1367] T d3d114f397e44d07a729f6eb09f279cc P 4325d7d0ed484ad78a6ee54e9a003431: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
I20240503 02:16:16.059201 18944 tablet_bootstrap.cc:492] T a8cc6cfa2b8240c9981a692e0896ef50 P 0f3838ee7ad04e94aa65ac750dad0833: No bootstrap required, opened a new log
I20240503 02:16:16.059242 18943 raft_consensus.cc:348] T d3d114f397e44d07a729f6eb09f279cc P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:16.059428 18944 ts_tablet_manager.cc:1367] T a8cc6cfa2b8240c9981a692e0896ef50 P 0f3838ee7ad04e94aa65ac750dad0833: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 02:16:16.059506 18943 raft_consensus.cc:374] T d3d114f397e44d07a729f6eb09f279cc P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:16.059602 18943 raft_consensus.cc:729] T d3d114f397e44d07a729f6eb09f279cc P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4325d7d0ed484ad78a6ee54e9a003431, State: Initialized, Role: FOLLOWER
I20240503 02:16:16.059947 18943 consensus_queue.cc:260] T d3d114f397e44d07a729f6eb09f279cc P 4325d7d0ed484ad78a6ee54e9a003431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:16.060523 18943 ts_tablet_manager.cc:1398] T d3d114f397e44d07a729f6eb09f279cc P 4325d7d0ed484ad78a6ee54e9a003431: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
I20240503 02:16:16.060441 18944 raft_consensus.cc:348] T a8cc6cfa2b8240c9981a692e0896ef50 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:16.060747 18944 raft_consensus.cc:374] T a8cc6cfa2b8240c9981a692e0896ef50 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:16.060885 18943 tablet_bootstrap.cc:492] T e53bab9a159147d79a1f96269f1c1201 P 4325d7d0ed484ad78a6ee54e9a003431: Bootstrap starting.
I20240503 02:16:16.060873 18944 raft_consensus.cc:729] T a8cc6cfa2b8240c9981a692e0896ef50 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Initialized, Role: FOLLOWER
I20240503 02:16:16.061179 18942 tablet_bootstrap.cc:492] T e53bab9a159147d79a1f96269f1c1201 P 836b43a7ee9146948319b8d37434b630: No bootstrap required, opened a new log
I20240503 02:16:16.061206 18944 consensus_queue.cc:260] T a8cc6cfa2b8240c9981a692e0896ef50 P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:16.061381 18942 ts_tablet_manager.cc:1367] T e53bab9a159147d79a1f96269f1c1201 P 836b43a7ee9146948319b8d37434b630: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 02:16:16.061753 18944 ts_tablet_manager.cc:1398] T a8cc6cfa2b8240c9981a692e0896ef50 P 0f3838ee7ad04e94aa65ac750dad0833: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:16:16.062448 18942 raft_consensus.cc:348] T e53bab9a159147d79a1f96269f1c1201 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:16.062734 18942 raft_consensus.cc:374] T e53bab9a159147d79a1f96269f1c1201 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:16.062875 18942 raft_consensus.cc:729] T e53bab9a159147d79a1f96269f1c1201 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Initialized, Role: FOLLOWER
I20240503 02:16:16.063257 18942 consensus_queue.cc:260] T e53bab9a159147d79a1f96269f1c1201 P 836b43a7ee9146948319b8d37434b630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:16.064074 18942 ts_tablet_manager.cc:1398] T e53bab9a159147d79a1f96269f1c1201 P 836b43a7ee9146948319b8d37434b630: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:16:16.064708 18943 tablet_bootstrap.cc:654] T e53bab9a159147d79a1f96269f1c1201 P 4325d7d0ed484ad78a6ee54e9a003431: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:16.064965 18942 tablet_bootstrap.cc:492] T d3d114f397e44d07a729f6eb09f279cc P 836b43a7ee9146948319b8d37434b630: Bootstrap starting.
I20240503 02:16:16.067147 18942 tablet_bootstrap.cc:654] T d3d114f397e44d07a729f6eb09f279cc P 836b43a7ee9146948319b8d37434b630: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:16.072058 18942 tablet_bootstrap.cc:492] T d3d114f397e44d07a729f6eb09f279cc P 836b43a7ee9146948319b8d37434b630: No bootstrap required, opened a new log
I20240503 02:16:16.072055 18946 tablet_bootstrap.cc:492] T e53bab9a159147d79a1f96269f1c1201 P 1cd8517d70d94ae494e713df7eb61b12: No bootstrap required, opened a new log
I20240503 02:16:16.072206 18946 ts_tablet_manager.cc:1367] T e53bab9a159147d79a1f96269f1c1201 P 1cd8517d70d94ae494e713df7eb61b12: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.007s
I20240503 02:16:16.072202 18942 ts_tablet_manager.cc:1367] T d3d114f397e44d07a729f6eb09f279cc P 836b43a7ee9146948319b8d37434b630: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240503 02:16:16.072844 18943 tablet_bootstrap.cc:492] T e53bab9a159147d79a1f96269f1c1201 P 4325d7d0ed484ad78a6ee54e9a003431: No bootstrap required, opened a new log
I20240503 02:16:16.072981 18943 ts_tablet_manager.cc:1367] T e53bab9a159147d79a1f96269f1c1201 P 4325d7d0ed484ad78a6ee54e9a003431: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20240503 02:16:16.072976 18946 raft_consensus.cc:348] T e53bab9a159147d79a1f96269f1c1201 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:16.072962 18942 raft_consensus.cc:348] T d3d114f397e44d07a729f6eb09f279cc P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:16.073199 18942 raft_consensus.cc:374] T d3d114f397e44d07a729f6eb09f279cc P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:16.073199 18946 raft_consensus.cc:374] T e53bab9a159147d79a1f96269f1c1201 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:16.073295 18942 raft_consensus.cc:729] T d3d114f397e44d07a729f6eb09f279cc P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Initialized, Role: FOLLOWER
I20240503 02:16:16.073302 18946 raft_consensus.cc:729] T e53bab9a159147d79a1f96269f1c1201 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Initialized, Role: FOLLOWER
I20240503 02:16:16.073541 18942 consensus_queue.cc:260] T d3d114f397e44d07a729f6eb09f279cc P 836b43a7ee9146948319b8d37434b630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:16.073639 18943 raft_consensus.cc:348] T e53bab9a159147d79a1f96269f1c1201 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:16.073848 18943 raft_consensus.cc:374] T e53bab9a159147d79a1f96269f1c1201 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:16.073601 18946 consensus_queue.cc:260] T e53bab9a159147d79a1f96269f1c1201 P 1cd8517d70d94ae494e713df7eb61b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:16.073945 18943 raft_consensus.cc:729] T e53bab9a159147d79a1f96269f1c1201 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4325d7d0ed484ad78a6ee54e9a003431, State: Initialized, Role: FOLLOWER
I20240503 02:16:16.074307 18946 ts_tablet_manager.cc:1398] T e53bab9a159147d79a1f96269f1c1201 P 1cd8517d70d94ae494e713df7eb61b12: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:16:16.074195 18943 consensus_queue.cc:260] T e53bab9a159147d79a1f96269f1c1201 P 4325d7d0ed484ad78a6ee54e9a003431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:16.074657 18942 ts_tablet_manager.cc:1398] T d3d114f397e44d07a729f6eb09f279cc P 836b43a7ee9146948319b8d37434b630: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:16:16.075595 18943 ts_tablet_manager.cc:1398] T e53bab9a159147d79a1f96269f1c1201 P 4325d7d0ed484ad78a6ee54e9a003431: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:16:16.150396 18850 raft_consensus.cc:482] T a8cc6cfa2b8240c9981a692e0896ef50 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:16.150616 18850 raft_consensus.cc:504] T a8cc6cfa2b8240c9981a692e0896ef50 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:16.151413 18850 leader_election.cc:288] T a8cc6cfa2b8240c9981a692e0896ef50 P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837), 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703)
I20240503 02:16:16.152772 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8cc6cfa2b8240c9981a692e0896ef50" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431" is_pre_election: true
I20240503 02:16:16.153069 18155 raft_consensus.cc:2454] T a8cc6cfa2b8240c9981a692e0896ef50 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 0.
I20240503 02:16:16.153541 17828 leader_election.cc:302] T a8cc6cfa2b8240c9981a692e0896ef50 P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 4325d7d0ed484ad78a6ee54e9a003431; no voters:
I20240503 02:16:16.153793 18850 raft_consensus.cc:2781] T a8cc6cfa2b8240c9981a692e0896ef50 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:16:16.153887 18850 raft_consensus.cc:482] T a8cc6cfa2b8240c9981a692e0896ef50 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:16:16.153975 18850 raft_consensus.cc:3037] T a8cc6cfa2b8240c9981a692e0896ef50 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:16.156234 18850 raft_consensus.cc:504] T a8cc6cfa2b8240c9981a692e0896ef50 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:16.156922 18850 leader_election.cc:288] T a8cc6cfa2b8240c9981a692e0896ef50 P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 1 election: Requested vote from peers 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837), 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703)
I20240503 02:16:16.157481 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8cc6cfa2b8240c9981a692e0896ef50" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431"
I20240503 02:16:16.157649 18155 raft_consensus.cc:3037] T a8cc6cfa2b8240c9981a692e0896ef50 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:16.158042 17763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8cc6cfa2b8240c9981a692e0896ef50" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836b43a7ee9146948319b8d37434b630" is_pre_election: true
I20240503 02:16:16.158164 17764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8cc6cfa2b8240c9981a692e0896ef50" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836b43a7ee9146948319b8d37434b630"
I20240503 02:16:16.158267 17763 raft_consensus.cc:2454] T a8cc6cfa2b8240c9981a692e0896ef50 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 0.
I20240503 02:16:16.159631 18155 raft_consensus.cc:2454] T a8cc6cfa2b8240c9981a692e0896ef50 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 1.
I20240503 02:16:16.159976 17828 leader_election.cc:302] T a8cc6cfa2b8240c9981a692e0896ef50 P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 4325d7d0ed484ad78a6ee54e9a003431; no voters: 836b43a7ee9146948319b8d37434b630
I20240503 02:16:16.160280 18850 raft_consensus.cc:2781] T a8cc6cfa2b8240c9981a692e0896ef50 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:16:16.160406 18850 raft_consensus.cc:686] T a8cc6cfa2b8240c9981a692e0896ef50 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 LEADER]: Becoming Leader. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:16.160625 18850 consensus_queue.cc:237] T a8cc6cfa2b8240c9981a692e0896ef50 P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:16.163141 17619 catalog_manager.cc:5484] T a8cc6cfa2b8240c9981a692e0896ef50 P 0f3838ee7ad04e94aa65ac750dad0833 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2). New cstate: current_term: 1 leader_uuid: "0f3838ee7ad04e94aa65ac750dad0833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } }
I20240503 02:16:16.187697 18826 raft_consensus.cc:482] T e53bab9a159147d79a1f96269f1c1201 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:16.187860 18826 raft_consensus.cc:504] T e53bab9a159147d79a1f96269f1c1201 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:16.188606 18826 leader_election.cc:288] T e53bab9a159147d79a1f96269f1c1201 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703), 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759)
I20240503 02:16:16.189144 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e53bab9a159147d79a1f96269f1c1201" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431" is_pre_election: true
I20240503 02:16:16.189466 18155 raft_consensus.cc:2454] T e53bab9a159147d79a1f96269f1c1201 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 0.
I20240503 02:16:16.189893 17697 leader_election.cc:302] T e53bab9a159147d79a1f96269f1c1201 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4325d7d0ed484ad78a6ee54e9a003431, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:16.190189 18826 raft_consensus.cc:2781] T e53bab9a159147d79a1f96269f1c1201 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:16:16.190346 18826 raft_consensus.cc:482] T e53bab9a159147d79a1f96269f1c1201 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:16:16.190471 18826 raft_consensus.cc:3037] T e53bab9a159147d79a1f96269f1c1201 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:16.193161 18826 raft_consensus.cc:504] T e53bab9a159147d79a1f96269f1c1201 P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:16.193951 18826 leader_election.cc:288] T e53bab9a159147d79a1f96269f1c1201 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 1 election: Requested vote from peers 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703), 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759)
I20240503 02:16:16.194337 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e53bab9a159147d79a1f96269f1c1201" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431"
I20240503 02:16:16.194522 18155 raft_consensus.cc:3037] T e53bab9a159147d79a1f96269f1c1201 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:16.195767 18025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e53bab9a159147d79a1f96269f1c1201" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd8517d70d94ae494e713df7eb61b12" is_pre_election: true
I20240503 02:16:16.195832 18023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e53bab9a159147d79a1f96269f1c1201" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:16.196053 18025 raft_consensus.cc:2454] T e53bab9a159147d79a1f96269f1c1201 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 0.
I20240503 02:16:16.196776 18155 raft_consensus.cc:2454] T e53bab9a159147d79a1f96269f1c1201 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 1.
I20240503 02:16:16.197160 17697 leader_election.cc:302] T e53bab9a159147d79a1f96269f1c1201 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4325d7d0ed484ad78a6ee54e9a003431, 836b43a7ee9146948319b8d37434b630; no voters: 1cd8517d70d94ae494e713df7eb61b12
I20240503 02:16:16.197464 18826 raft_consensus.cc:2781] T e53bab9a159147d79a1f96269f1c1201 P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:16:16.197829 18826 raft_consensus.cc:686] T e53bab9a159147d79a1f96269f1c1201 P 836b43a7ee9146948319b8d37434b630 [term 1 LEADER]: Becoming Leader. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Running, Role: LEADER
I20240503 02:16:16.198136 18826 consensus_queue.cc:237] T e53bab9a159147d79a1f96269f1c1201 P 836b43a7ee9146948319b8d37434b630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:16.200673 17619 catalog_manager.cc:5484] T e53bab9a159147d79a1f96269f1c1201 P 836b43a7ee9146948319b8d37434b630 reported cstate change: term changed from 0 to 1, leader changed from <none> to 836b43a7ee9146948319b8d37434b630 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "836b43a7ee9146948319b8d37434b630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } }
I20240503 02:16:16.301230 18868 raft_consensus.cc:482] T d3d114f397e44d07a729f6eb09f279cc P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:16.301407 18868 raft_consensus.cc:504] T d3d114f397e44d07a729f6eb09f279cc P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:16.302119 18868 leader_election.cc:288] T d3d114f397e44d07a729f6eb09f279cc P 4325d7d0ed484ad78a6ee54e9a003431 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647), 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837)
I20240503 02:16:16.302637 17895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3d114f397e44d07a729f6eb09f279cc" candidate_uuid: "4325d7d0ed484ad78a6ee54e9a003431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833" is_pre_election: true
I20240503 02:16:16.302968 17895 raft_consensus.cc:2454] T d3d114f397e44d07a729f6eb09f279cc P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4325d7d0ed484ad78a6ee54e9a003431 in term 0.
I20240503 02:16:16.303465 18088 leader_election.cc:302] T d3d114f397e44d07a729f6eb09f279cc P 4325d7d0ed484ad78a6ee54e9a003431 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 4325d7d0ed484ad78a6ee54e9a003431; no voters:
I20240503 02:16:16.303773 18868 raft_consensus.cc:2781] T d3d114f397e44d07a729f6eb09f279cc P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:16:16.303920 18868 raft_consensus.cc:482] T d3d114f397e44d07a729f6eb09f279cc P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:16:16.304023 18868 raft_consensus.cc:3037] T d3d114f397e44d07a729f6eb09f279cc P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:16.305948 18868 raft_consensus.cc:504] T d3d114f397e44d07a729f6eb09f279cc P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:16.306448 18868 leader_election.cc:288] T d3d114f397e44d07a729f6eb09f279cc P 4325d7d0ed484ad78a6ee54e9a003431 [CANDIDATE]: Term 1 election: Requested vote from peers 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647), 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837)
I20240503 02:16:16.306818 17894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3d114f397e44d07a729f6eb09f279cc" candidate_uuid: "4325d7d0ed484ad78a6ee54e9a003431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
I20240503 02:16:16.306998 17894 raft_consensus.cc:3037] T d3d114f397e44d07a729f6eb09f279cc P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:16.308507 17764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3d114f397e44d07a729f6eb09f279cc" candidate_uuid: "4325d7d0ed484ad78a6ee54e9a003431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836b43a7ee9146948319b8d37434b630" is_pre_election: true
I20240503 02:16:16.308650 17763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3d114f397e44d07a729f6eb09f279cc" candidate_uuid: "4325d7d0ed484ad78a6ee54e9a003431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836b43a7ee9146948319b8d37434b630"
I20240503 02:16:16.308773 17764 raft_consensus.cc:2454] T d3d114f397e44d07a729f6eb09f279cc P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4325d7d0ed484ad78a6ee54e9a003431 in term 0.
I20240503 02:16:16.309206 17894 raft_consensus.cc:2454] T d3d114f397e44d07a729f6eb09f279cc P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4325d7d0ed484ad78a6ee54e9a003431 in term 1.
I20240503 02:16:16.309523 18088 leader_election.cc:302] T d3d114f397e44d07a729f6eb09f279cc P 4325d7d0ed484ad78a6ee54e9a003431 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 4325d7d0ed484ad78a6ee54e9a003431; no voters: 836b43a7ee9146948319b8d37434b630
I20240503 02:16:16.309818 18868 raft_consensus.cc:2781] T d3d114f397e44d07a729f6eb09f279cc P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:16:16.309952 18868 raft_consensus.cc:686] T d3d114f397e44d07a729f6eb09f279cc P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 LEADER]: Becoming Leader. State: Replica: 4325d7d0ed484ad78a6ee54e9a003431, State: Running, Role: LEADER
I20240503 02:16:16.310209 18868 consensus_queue.cc:237] T d3d114f397e44d07a729f6eb09f279cc P 4325d7d0ed484ad78a6ee54e9a003431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:16.312803 17619 catalog_manager.cc:5484] T d3d114f397e44d07a729f6eb09f279cc P 4325d7d0ed484ad78a6ee54e9a003431 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4). New cstate: current_term: 1 leader_uuid: "4325d7d0ed484ad78a6ee54e9a003431" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } }
I20240503 02:16:16.548487 17763 raft_consensus.cc:3037] T a8cc6cfa2b8240c9981a692e0896ef50 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:16.551330 18850 consensus_queue.cc:1035] T a8cc6cfa2b8240c9981a692e0896ef50 P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:16:16.564623 18850 consensus_queue.cc:1035] T a8cc6cfa2b8240c9981a692e0896ef50 P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:16:16.658833 18963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:16:16.659188 18963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:16:16.666663 18963 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:16:16.693825 18025 raft_consensus.cc:3037] T e53bab9a159147d79a1f96269f1c1201 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Advancing to term 1
Using table 'rebalance_test_table_2'
I20240503 02:16:16.697376 18826 consensus_queue.cc:1035] T e53bab9a159147d79a1f96269f1c1201 P 836b43a7ee9146948319b8d37434b630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:16:16.702351 18985 consensus_queue.cc:1035] T e53bab9a159147d79a1f96269f1c1201 P 836b43a7ee9146948319b8d37434b630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:16:16.735561 18868 consensus_queue.cc:1035] T d3d114f397e44d07a729f6eb09f279cc P 4325d7d0ed484ad78a6ee54e9a003431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:16:16.746841 17763 raft_consensus.cc:3037] T d3d114f397e44d07a729f6eb09f279cc P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:16.751586 18855 consensus_queue.cc:1035] T d3d114f397e44d07a729f6eb09f279cc P 4325d7d0ed484ad78a6ee54e9a003431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:16:16.977813 18088 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.108.2:41647, user_credentials={real_user=slave}} blocked reactor thread for 68277us
INSERT report
rows total: 2000
time total: 316.444 ms
time per row: 0.158222 ms
I20240503 02:16:17.140846 17622 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44922:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:16:17.149431 17742 tablet_service.cc:1467] Processing CreateTablet for tablet 7c160a83bd434c56ab99f86d6df68317 (DEFAULT_TABLE table=rebalance_test_table_3 [id=94162ee58ce442a8a225b1cf00e6c2ce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:17.149674 17875 tablet_service.cc:1467] Processing CreateTablet for tablet 7c160a83bd434c56ab99f86d6df68317 (DEFAULT_TABLE table=rebalance_test_table_3 [id=94162ee58ce442a8a225b1cf00e6c2ce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:17.150002 17742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c160a83bd434c56ab99f86d6df68317. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:17.150106 17875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c160a83bd434c56ab99f86d6df68317. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:17.151615 18135 tablet_service.cc:1467] Processing CreateTablet for tablet 18a22320e5ba41c980e95c1a8d4cf99a (DEFAULT_TABLE table=rebalance_test_table_3 [id=94162ee58ce442a8a225b1cf00e6c2ce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:17.152036 18135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18a22320e5ba41c980e95c1a8d4cf99a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:17.151968 18003 tablet_service.cc:1467] Processing CreateTablet for tablet 2625f9a7afe74874bd6ccb05d6d6e673 (DEFAULT_TABLE table=rebalance_test_table_3 [id=94162ee58ce442a8a225b1cf00e6c2ce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:17.152405 18003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2625f9a7afe74874bd6ccb05d6d6e673. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:17.156767 18005 tablet_service.cc:1467] Processing CreateTablet for tablet 18a22320e5ba41c980e95c1a8d4cf99a (DEFAULT_TABLE table=rebalance_test_table_3 [id=94162ee58ce442a8a225b1cf00e6c2ce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:17.157279 18005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18a22320e5ba41c980e95c1a8d4cf99a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:17.157536 18133 tablet_service.cc:1467] Processing CreateTablet for tablet 2625f9a7afe74874bd6ccb05d6d6e673 (DEFAULT_TABLE table=rebalance_test_table_3 [id=94162ee58ce442a8a225b1cf00e6c2ce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:17.158013 18133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2625f9a7afe74874bd6ccb05d6d6e673. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:17.161500 18004 tablet_service.cc:1467] Processing CreateTablet for tablet 7c160a83bd434c56ab99f86d6df68317 (DEFAULT_TABLE table=rebalance_test_table_3 [id=94162ee58ce442a8a225b1cf00e6c2ce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:17.161864 18004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c160a83bd434c56ab99f86d6df68317. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:17.161990 17874 tablet_service.cc:1467] Processing CreateTablet for tablet 2625f9a7afe74874bd6ccb05d6d6e673 (DEFAULT_TABLE table=rebalance_test_table_3 [id=94162ee58ce442a8a225b1cf00e6c2ce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:17.162230 17743 tablet_service.cc:1467] Processing CreateTablet for tablet 18a22320e5ba41c980e95c1a8d4cf99a (DEFAULT_TABLE table=rebalance_test_table_3 [id=94162ee58ce442a8a225b1cf00e6c2ce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:17.162345 17874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2625f9a7afe74874bd6ccb05d6d6e673. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:17.162596 17743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18a22320e5ba41c980e95c1a8d4cf99a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:17.170703 19028 tablet_bootstrap.cc:492] T 18a22320e5ba41c980e95c1a8d4cf99a P 4325d7d0ed484ad78a6ee54e9a003431: Bootstrap starting.
I20240503 02:16:17.171641 19026 tablet_bootstrap.cc:492] T 7c160a83bd434c56ab99f86d6df68317 P 836b43a7ee9146948319b8d37434b630: Bootstrap starting.
I20240503 02:16:17.173247 19028 tablet_bootstrap.cc:654] T 18a22320e5ba41c980e95c1a8d4cf99a P 4325d7d0ed484ad78a6ee54e9a003431: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:17.173411 19026 tablet_bootstrap.cc:654] T 7c160a83bd434c56ab99f86d6df68317 P 836b43a7ee9146948319b8d37434b630: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:17.174158 19029 tablet_bootstrap.cc:492] T 18a22320e5ba41c980e95c1a8d4cf99a P 1cd8517d70d94ae494e713df7eb61b12: Bootstrap starting.
I20240503 02:16:17.175884 19029 tablet_bootstrap.cc:654] T 18a22320e5ba41c980e95c1a8d4cf99a P 1cd8517d70d94ae494e713df7eb61b12: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:17.178496 19028 tablet_bootstrap.cc:492] T 18a22320e5ba41c980e95c1a8d4cf99a P 4325d7d0ed484ad78a6ee54e9a003431: No bootstrap required, opened a new log
I20240503 02:16:17.178658 19028 ts_tablet_manager.cc:1367] T 18a22320e5ba41c980e95c1a8d4cf99a P 4325d7d0ed484ad78a6ee54e9a003431: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240503 02:16:17.179504 19026 tablet_bootstrap.cc:492] T 7c160a83bd434c56ab99f86d6df68317 P 836b43a7ee9146948319b8d37434b630: No bootstrap required, opened a new log
I20240503 02:16:17.179661 19026 ts_tablet_manager.cc:1367] T 7c160a83bd434c56ab99f86d6df68317 P 836b43a7ee9146948319b8d37434b630: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240503 02:16:17.180048 19029 tablet_bootstrap.cc:492] T 18a22320e5ba41c980e95c1a8d4cf99a P 1cd8517d70d94ae494e713df7eb61b12: No bootstrap required, opened a new log
I20240503 02:16:17.180197 19029 ts_tablet_manager.cc:1367] T 18a22320e5ba41c980e95c1a8d4cf99a P 1cd8517d70d94ae494e713df7eb61b12: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20240503 02:16:17.180023 19028 raft_consensus.cc:348] T 18a22320e5ba41c980e95c1a8d4cf99a P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:17.180487 19028 raft_consensus.cc:374] T 18a22320e5ba41c980e95c1a8d4cf99a P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:17.180500 19026 raft_consensus.cc:348] T 7c160a83bd434c56ab99f86d6df68317 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:17.180697 19026 raft_consensus.cc:374] T 7c160a83bd434c56ab99f86d6df68317 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:17.180644 19028 raft_consensus.cc:729] T 18a22320e5ba41c980e95c1a8d4cf99a P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4325d7d0ed484ad78a6ee54e9a003431, State: Initialized, Role: FOLLOWER
I20240503 02:16:17.180795 19026 raft_consensus.cc:729] T 7c160a83bd434c56ab99f86d6df68317 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Initialized, Role: FOLLOWER
I20240503 02:16:17.180999 19029 raft_consensus.cc:348] T 18a22320e5ba41c980e95c1a8d4cf99a P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:17.181064 19026 consensus_queue.cc:260] T 7c160a83bd434c56ab99f86d6df68317 P 836b43a7ee9146948319b8d37434b630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:17.181212 19029 raft_consensus.cc:374] T 18a22320e5ba41c980e95c1a8d4cf99a P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:17.181174 19028 consensus_queue.cc:260] T 18a22320e5ba41c980e95c1a8d4cf99a P 4325d7d0ed484ad78a6ee54e9a003431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:17.181627 19026 ts_tablet_manager.cc:1398] T 7c160a83bd434c56ab99f86d6df68317 P 836b43a7ee9146948319b8d37434b630: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:16:17.181928 19026 tablet_bootstrap.cc:492] T 18a22320e5ba41c980e95c1a8d4cf99a P 836b43a7ee9146948319b8d37434b630: Bootstrap starting.
I20240503 02:16:17.182739 19028 ts_tablet_manager.cc:1398] T 18a22320e5ba41c980e95c1a8d4cf99a P 4325d7d0ed484ad78a6ee54e9a003431: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:16:17.183120 19028 tablet_bootstrap.cc:492] T 2625f9a7afe74874bd6ccb05d6d6e673 P 4325d7d0ed484ad78a6ee54e9a003431: Bootstrap starting.
I20240503 02:16:17.183490 19026 tablet_bootstrap.cc:654] T 18a22320e5ba41c980e95c1a8d4cf99a P 836b43a7ee9146948319b8d37434b630: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:17.185387 19028 tablet_bootstrap.cc:654] T 2625f9a7afe74874bd6ccb05d6d6e673 P 4325d7d0ed484ad78a6ee54e9a003431: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:17.186971 19027 tablet_bootstrap.cc:492] T 7c160a83bd434c56ab99f86d6df68317 P 0f3838ee7ad04e94aa65ac750dad0833: Bootstrap starting.
I20240503 02:16:17.188484 19026 tablet_bootstrap.cc:492] T 18a22320e5ba41c980e95c1a8d4cf99a P 836b43a7ee9146948319b8d37434b630: No bootstrap required, opened a new log
I20240503 02:16:17.188625 19027 tablet_bootstrap.cc:654] T 7c160a83bd434c56ab99f86d6df68317 P 0f3838ee7ad04e94aa65ac750dad0833: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:17.188647 19026 ts_tablet_manager.cc:1367] T 18a22320e5ba41c980e95c1a8d4cf99a P 836b43a7ee9146948319b8d37434b630: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20240503 02:16:17.189498 19026 raft_consensus.cc:348] T 18a22320e5ba41c980e95c1a8d4cf99a P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:17.189761 19026 raft_consensus.cc:374] T 18a22320e5ba41c980e95c1a8d4cf99a P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:17.189855 19026 raft_consensus.cc:729] T 18a22320e5ba41c980e95c1a8d4cf99a P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Initialized, Role: FOLLOWER
I20240503 02:16:17.189957 19029 raft_consensus.cc:729] T 18a22320e5ba41c980e95c1a8d4cf99a P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Initialized, Role: FOLLOWER
I20240503 02:16:17.190089 19026 consensus_queue.cc:260] T 18a22320e5ba41c980e95c1a8d4cf99a P 836b43a7ee9146948319b8d37434b630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:17.190315 19028 tablet_bootstrap.cc:492] T 2625f9a7afe74874bd6ccb05d6d6e673 P 4325d7d0ed484ad78a6ee54e9a003431: No bootstrap required, opened a new log
I20240503 02:16:17.190225 19029 consensus_queue.cc:260] T 18a22320e5ba41c980e95c1a8d4cf99a P 1cd8517d70d94ae494e713df7eb61b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:17.190459 19028 ts_tablet_manager.cc:1367] T 2625f9a7afe74874bd6ccb05d6d6e673 P 4325d7d0ed484ad78a6ee54e9a003431: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240503 02:16:17.190567 19026 ts_tablet_manager.cc:1398] T 18a22320e5ba41c980e95c1a8d4cf99a P 836b43a7ee9146948319b8d37434b630: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:16:17.191329 19028 raft_consensus.cc:348] T 2625f9a7afe74874bd6ccb05d6d6e673 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:17.191433 19029 ts_tablet_manager.cc:1398] T 18a22320e5ba41c980e95c1a8d4cf99a P 1cd8517d70d94ae494e713df7eb61b12: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20240503 02:16:17.191553 19028 raft_consensus.cc:374] T 2625f9a7afe74874bd6ccb05d6d6e673 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:17.191645 19028 raft_consensus.cc:729] T 2625f9a7afe74874bd6ccb05d6d6e673 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4325d7d0ed484ad78a6ee54e9a003431, State: Initialized, Role: FOLLOWER
I20240503 02:16:17.191784 19029 tablet_bootstrap.cc:492] T 7c160a83bd434c56ab99f86d6df68317 P 1cd8517d70d94ae494e713df7eb61b12: Bootstrap starting.
I20240503 02:16:17.191912 19028 consensus_queue.cc:260] T 2625f9a7afe74874bd6ccb05d6d6e673 P 4325d7d0ed484ad78a6ee54e9a003431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:17.192426 19027 tablet_bootstrap.cc:492] T 7c160a83bd434c56ab99f86d6df68317 P 0f3838ee7ad04e94aa65ac750dad0833: No bootstrap required, opened a new log
I20240503 02:16:17.192530 19028 ts_tablet_manager.cc:1398] T 2625f9a7afe74874bd6ccb05d6d6e673 P 4325d7d0ed484ad78a6ee54e9a003431: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:16:17.192625 19027 ts_tablet_manager.cc:1367] T 7c160a83bd434c56ab99f86d6df68317 P 0f3838ee7ad04e94aa65ac750dad0833: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 02:16:17.193426 19029 tablet_bootstrap.cc:654] T 7c160a83bd434c56ab99f86d6df68317 P 1cd8517d70d94ae494e713df7eb61b12: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:17.193620 19027 raft_consensus.cc:348] T 7c160a83bd434c56ab99f86d6df68317 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:17.193925 19027 raft_consensus.cc:374] T 7c160a83bd434c56ab99f86d6df68317 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:17.194047 19027 raft_consensus.cc:729] T 7c160a83bd434c56ab99f86d6df68317 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Initialized, Role: FOLLOWER
I20240503 02:16:17.194314 19027 consensus_queue.cc:260] T 7c160a83bd434c56ab99f86d6df68317 P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:17.195010 19027 ts_tablet_manager.cc:1398] T 7c160a83bd434c56ab99f86d6df68317 P 0f3838ee7ad04e94aa65ac750dad0833: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:16:17.195345 19027 tablet_bootstrap.cc:492] T 2625f9a7afe74874bd6ccb05d6d6e673 P 0f3838ee7ad04e94aa65ac750dad0833: Bootstrap starting.
I20240503 02:16:17.197523 19029 tablet_bootstrap.cc:492] T 7c160a83bd434c56ab99f86d6df68317 P 1cd8517d70d94ae494e713df7eb61b12: No bootstrap required, opened a new log
I20240503 02:16:17.197696 19029 ts_tablet_manager.cc:1367] T 7c160a83bd434c56ab99f86d6df68317 P 1cd8517d70d94ae494e713df7eb61b12: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20240503 02:16:17.197865 19027 tablet_bootstrap.cc:654] T 2625f9a7afe74874bd6ccb05d6d6e673 P 0f3838ee7ad04e94aa65ac750dad0833: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:17.198526 19029 raft_consensus.cc:348] T 7c160a83bd434c56ab99f86d6df68317 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:17.198742 19029 raft_consensus.cc:374] T 7c160a83bd434c56ab99f86d6df68317 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:17.198839 19029 raft_consensus.cc:729] T 7c160a83bd434c56ab99f86d6df68317 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Initialized, Role: FOLLOWER
I20240503 02:16:17.199182 19029 consensus_queue.cc:260] T 7c160a83bd434c56ab99f86d6df68317 P 1cd8517d70d94ae494e713df7eb61b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:17.200328 19029 ts_tablet_manager.cc:1398] T 7c160a83bd434c56ab99f86d6df68317 P 1cd8517d70d94ae494e713df7eb61b12: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:16:17.200448 19027 tablet_bootstrap.cc:492] T 2625f9a7afe74874bd6ccb05d6d6e673 P 0f3838ee7ad04e94aa65ac750dad0833: No bootstrap required, opened a new log
I20240503 02:16:17.200618 19027 ts_tablet_manager.cc:1367] T 2625f9a7afe74874bd6ccb05d6d6e673 P 0f3838ee7ad04e94aa65ac750dad0833: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:16:17.200631 19029 tablet_bootstrap.cc:492] T 2625f9a7afe74874bd6ccb05d6d6e673 P 1cd8517d70d94ae494e713df7eb61b12: Bootstrap starting.
I20240503 02:16:17.201361 19027 raft_consensus.cc:348] T 2625f9a7afe74874bd6ccb05d6d6e673 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:17.201608 19027 raft_consensus.cc:374] T 2625f9a7afe74874bd6ccb05d6d6e673 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:17.201716 19027 raft_consensus.cc:729] T 2625f9a7afe74874bd6ccb05d6d6e673 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Initialized, Role: FOLLOWER
I20240503 02:16:17.201975 19029 tablet_bootstrap.cc:654] T 2625f9a7afe74874bd6ccb05d6d6e673 P 1cd8517d70d94ae494e713df7eb61b12: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:17.201953 19027 consensus_queue.cc:260] T 2625f9a7afe74874bd6ccb05d6d6e673 P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:17.202489 19027 ts_tablet_manager.cc:1398] T 2625f9a7afe74874bd6ccb05d6d6e673 P 0f3838ee7ad04e94aa65ac750dad0833: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:16:17.203889 19029 tablet_bootstrap.cc:492] T 2625f9a7afe74874bd6ccb05d6d6e673 P 1cd8517d70d94ae494e713df7eb61b12: No bootstrap required, opened a new log
I20240503 02:16:17.204016 19029 ts_tablet_manager.cc:1367] T 2625f9a7afe74874bd6ccb05d6d6e673 P 1cd8517d70d94ae494e713df7eb61b12: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:16:17.204766 19029 raft_consensus.cc:348] T 2625f9a7afe74874bd6ccb05d6d6e673 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:17.204913 19029 raft_consensus.cc:374] T 2625f9a7afe74874bd6ccb05d6d6e673 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:17.205010 19029 raft_consensus.cc:729] T 2625f9a7afe74874bd6ccb05d6d6e673 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Initialized, Role: FOLLOWER
I20240503 02:16:17.205252 19029 consensus_queue.cc:260] T 2625f9a7afe74874bd6ccb05d6d6e673 P 1cd8517d70d94ae494e713df7eb61b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:17.205638 19029 ts_tablet_manager.cc:1398] T 2625f9a7afe74874bd6ccb05d6d6e673 P 1cd8517d70d94ae494e713df7eb61b12: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 02:16:17.228629 18868 raft_consensus.cc:482] T 2625f9a7afe74874bd6ccb05d6d6e673 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:17.228893 18868 raft_consensus.cc:504] T 2625f9a7afe74874bd6ccb05d6d6e673 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:17.229542 18868 leader_election.cc:288] T 2625f9a7afe74874bd6ccb05d6d6e673 P 4325d7d0ed484ad78a6ee54e9a003431 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647), 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759)
I20240503 02:16:17.230024 17895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2625f9a7afe74874bd6ccb05d6d6e673" candidate_uuid: "4325d7d0ed484ad78a6ee54e9a003431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833" is_pre_election: true
I20240503 02:16:17.230074 18025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2625f9a7afe74874bd6ccb05d6d6e673" candidate_uuid: "4325d7d0ed484ad78a6ee54e9a003431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd8517d70d94ae494e713df7eb61b12" is_pre_election: true
I20240503 02:16:17.230253 17895 raft_consensus.cc:2454] T 2625f9a7afe74874bd6ccb05d6d6e673 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4325d7d0ed484ad78a6ee54e9a003431 in term 0.
I20240503 02:16:17.230326 18025 raft_consensus.cc:2454] T 2625f9a7afe74874bd6ccb05d6d6e673 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4325d7d0ed484ad78a6ee54e9a003431 in term 0.
I20240503 02:16:17.230659 18088 leader_election.cc:302] T 2625f9a7afe74874bd6ccb05d6d6e673 P 4325d7d0ed484ad78a6ee54e9a003431 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 4325d7d0ed484ad78a6ee54e9a003431; no voters:
I20240503 02:16:17.230960 18868 raft_consensus.cc:2781] T 2625f9a7afe74874bd6ccb05d6d6e673 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:16:17.231060 18868 raft_consensus.cc:482] T 2625f9a7afe74874bd6ccb05d6d6e673 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:16:17.231154 18868 raft_consensus.cc:3037] T 2625f9a7afe74874bd6ccb05d6d6e673 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:17.233065 18868 raft_consensus.cc:504] T 2625f9a7afe74874bd6ccb05d6d6e673 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:17.233631 18868 leader_election.cc:288] T 2625f9a7afe74874bd6ccb05d6d6e673 P 4325d7d0ed484ad78a6ee54e9a003431 [CANDIDATE]: Term 1 election: Requested vote from peers 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647), 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759)
I20240503 02:16:17.234083 17895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2625f9a7afe74874bd6ccb05d6d6e673" candidate_uuid: "4325d7d0ed484ad78a6ee54e9a003431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
I20240503 02:16:17.234171 18025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2625f9a7afe74874bd6ccb05d6d6e673" candidate_uuid: "4325d7d0ed484ad78a6ee54e9a003431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:17.234319 17895 raft_consensus.cc:3037] T 2625f9a7afe74874bd6ccb05d6d6e673 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:17.234376 18025 raft_consensus.cc:3037] T 2625f9a7afe74874bd6ccb05d6d6e673 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:17.237175 18025 raft_consensus.cc:2454] T 2625f9a7afe74874bd6ccb05d6d6e673 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4325d7d0ed484ad78a6ee54e9a003431 in term 1.
I20240503 02:16:17.237241 17895 raft_consensus.cc:2454] T 2625f9a7afe74874bd6ccb05d6d6e673 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4325d7d0ed484ad78a6ee54e9a003431 in term 1.
I20240503 02:16:17.237741 18091 leader_election.cc:302] T 2625f9a7afe74874bd6ccb05d6d6e673 P 4325d7d0ed484ad78a6ee54e9a003431 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cd8517d70d94ae494e713df7eb61b12, 4325d7d0ed484ad78a6ee54e9a003431; no voters:
I20240503 02:16:17.238075 18868 raft_consensus.cc:2781] T 2625f9a7afe74874bd6ccb05d6d6e673 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:16:17.238235 18868 raft_consensus.cc:686] T 2625f9a7afe74874bd6ccb05d6d6e673 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 LEADER]: Becoming Leader. State: Replica: 4325d7d0ed484ad78a6ee54e9a003431, State: Running, Role: LEADER
I20240503 02:16:17.238508 18868 consensus_queue.cc:237] T 2625f9a7afe74874bd6ccb05d6d6e673 P 4325d7d0ed484ad78a6ee54e9a003431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:17.241231 17618 catalog_manager.cc:5484] T 2625f9a7afe74874bd6ccb05d6d6e673 P 4325d7d0ed484ad78a6ee54e9a003431 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4). New cstate: current_term: 1 leader_uuid: "4325d7d0ed484ad78a6ee54e9a003431" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } }
I20240503 02:16:17.264704 18826 raft_consensus.cc:482] T 7c160a83bd434c56ab99f86d6df68317 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:17.264884 18826 raft_consensus.cc:504] T 7c160a83bd434c56ab99f86d6df68317 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:17.265558 18826 leader_election.cc:288] T 7c160a83bd434c56ab99f86d6df68317 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647), 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759)
I20240503 02:16:17.266122 17895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c160a83bd434c56ab99f86d6df68317" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833" is_pre_election: true
I20240503 02:16:17.266158 18025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c160a83bd434c56ab99f86d6df68317" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd8517d70d94ae494e713df7eb61b12" is_pre_election: true
I20240503 02:16:17.266351 17895 raft_consensus.cc:2454] T 7c160a83bd434c56ab99f86d6df68317 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 0.
I20240503 02:16:17.266356 18025 raft_consensus.cc:2454] T 7c160a83bd434c56ab99f86d6df68317 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 0.
I20240503 02:16:17.266752 17697 leader_election.cc:302] T 7c160a83bd434c56ab99f86d6df68317 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:17.267057 18826 raft_consensus.cc:2781] T 7c160a83bd434c56ab99f86d6df68317 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:16:17.267163 18826 raft_consensus.cc:482] T 7c160a83bd434c56ab99f86d6df68317 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:16:17.267231 18826 raft_consensus.cc:3037] T 7c160a83bd434c56ab99f86d6df68317 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:17.269140 18826 raft_consensus.cc:504] T 7c160a83bd434c56ab99f86d6df68317 P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:17.269686 18826 leader_election.cc:288] T 7c160a83bd434c56ab99f86d6df68317 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 1 election: Requested vote from peers 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647), 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759)
I20240503 02:16:17.270153 17895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c160a83bd434c56ab99f86d6df68317" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
I20240503 02:16:17.270231 18025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c160a83bd434c56ab99f86d6df68317" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:17.270366 17895 raft_consensus.cc:3037] T 7c160a83bd434c56ab99f86d6df68317 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:17.270416 18025 raft_consensus.cc:3037] T 7c160a83bd434c56ab99f86d6df68317 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:17.273247 17895 raft_consensus.cc:2454] T 7c160a83bd434c56ab99f86d6df68317 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 1.
I20240503 02:16:17.273248 18025 raft_consensus.cc:2454] T 7c160a83bd434c56ab99f86d6df68317 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 1.
I20240503 02:16:17.273622 17697 leader_election.cc:302] T 7c160a83bd434c56ab99f86d6df68317 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:17.273998 18826 raft_consensus.cc:2781] T 7c160a83bd434c56ab99f86d6df68317 P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:16:17.274137 18826 raft_consensus.cc:686] T 7c160a83bd434c56ab99f86d6df68317 P 836b43a7ee9146948319b8d37434b630 [term 1 LEADER]: Becoming Leader. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Running, Role: LEADER
I20240503 02:16:17.274428 18826 consensus_queue.cc:237] T 7c160a83bd434c56ab99f86d6df68317 P 836b43a7ee9146948319b8d37434b630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:17.276933 17618 catalog_manager.cc:5484] T 7c160a83bd434c56ab99f86d6df68317 P 836b43a7ee9146948319b8d37434b630 reported cstate change: term changed from 0 to 1, leader changed from <none> to 836b43a7ee9146948319b8d37434b630 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "836b43a7ee9146948319b8d37434b630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } }
I20240503 02:16:17.514570 19041 raft_consensus.cc:482] T 18a22320e5ba41c980e95c1a8d4cf99a P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:17.514784 19041 raft_consensus.cc:504] T 18a22320e5ba41c980e95c1a8d4cf99a P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:17.515462 19041 leader_election.cc:288] T 18a22320e5ba41c980e95c1a8d4cf99a P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703), 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837)
I20240503 02:16:17.516012 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a22320e5ba41c980e95c1a8d4cf99a" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431" is_pre_election: true
I20240503 02:16:17.516027 17764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a22320e5ba41c980e95c1a8d4cf99a" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836b43a7ee9146948319b8d37434b630" is_pre_election: true
I20240503 02:16:17.516229 18155 raft_consensus.cc:2454] T 18a22320e5ba41c980e95c1a8d4cf99a P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cd8517d70d94ae494e713df7eb61b12 in term 0.
I20240503 02:16:17.516240 17764 raft_consensus.cc:2454] T 18a22320e5ba41c980e95c1a8d4cf99a P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cd8517d70d94ae494e713df7eb61b12 in term 0.
I20240503 02:16:17.516660 17958 leader_election.cc:302] T 18a22320e5ba41c980e95c1a8d4cf99a P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cd8517d70d94ae494e713df7eb61b12, 4325d7d0ed484ad78a6ee54e9a003431; no voters:
I20240503 02:16:17.516928 19041 raft_consensus.cc:2781] T 18a22320e5ba41c980e95c1a8d4cf99a P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:16:17.517246 19041 raft_consensus.cc:482] T 18a22320e5ba41c980e95c1a8d4cf99a P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:16:17.517380 19041 raft_consensus.cc:3037] T 18a22320e5ba41c980e95c1a8d4cf99a P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:17.519500 19041 raft_consensus.cc:504] T 18a22320e5ba41c980e95c1a8d4cf99a P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:17.520138 19041 leader_election.cc:288] T 18a22320e5ba41c980e95c1a8d4cf99a P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 1 election: Requested vote from peers 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703), 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837)
I20240503 02:16:17.520634 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a22320e5ba41c980e95c1a8d4cf99a" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431"
I20240503 02:16:17.520642 17764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a22320e5ba41c980e95c1a8d4cf99a" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836b43a7ee9146948319b8d37434b630"
I20240503 02:16:17.520874 17764 raft_consensus.cc:3037] T 18a22320e5ba41c980e95c1a8d4cf99a P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:17.520872 18155 raft_consensus.cc:3037] T 18a22320e5ba41c980e95c1a8d4cf99a P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:17.523665 18155 raft_consensus.cc:2454] T 18a22320e5ba41c980e95c1a8d4cf99a P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cd8517d70d94ae494e713df7eb61b12 in term 1.
I20240503 02:16:17.523705 17764 raft_consensus.cc:2454] T 18a22320e5ba41c980e95c1a8d4cf99a P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cd8517d70d94ae494e713df7eb61b12 in term 1.
I20240503 02:16:17.524077 17958 leader_election.cc:302] T 18a22320e5ba41c980e95c1a8d4cf99a P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cd8517d70d94ae494e713df7eb61b12, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:17.524353 19041 raft_consensus.cc:2781] T 18a22320e5ba41c980e95c1a8d4cf99a P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:16:17.524497 19041 raft_consensus.cc:686] T 18a22320e5ba41c980e95c1a8d4cf99a P 1cd8517d70d94ae494e713df7eb61b12 [term 1 LEADER]: Becoming Leader. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Running, Role: LEADER
I20240503 02:16:17.524757 19041 consensus_queue.cc:237] T 18a22320e5ba41c980e95c1a8d4cf99a P 1cd8517d70d94ae494e713df7eb61b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:17.527508 17618 catalog_manager.cc:5484] T 18a22320e5ba41c980e95c1a8d4cf99a P 1cd8517d70d94ae494e713df7eb61b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3). New cstate: current_term: 1 leader_uuid: "1cd8517d70d94ae494e713df7eb61b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } }
I20240503 02:16:17.645522 18868 consensus_queue.cc:1035] T 2625f9a7afe74874bd6ccb05d6d6e673 P 4325d7d0ed484ad78a6ee54e9a003431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:16:17.650182 18868 consensus_queue.cc:1035] T 2625f9a7afe74874bd6ccb05d6d6e673 P 4325d7d0ed484ad78a6ee54e9a003431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:16:17.742807 18826 consensus_queue.cc:1035] T 7c160a83bd434c56ab99f86d6df68317 P 836b43a7ee9146948319b8d37434b630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:16:17.747885 18826 consensus_queue.cc:1035] T 7c160a83bd434c56ab99f86d6df68317 P 836b43a7ee9146948319b8d37434b630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:16:17.878175 19043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:16:17.878468 19043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:16:17.885810 19043 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 02:16:17.987092 19041 consensus_queue.cc:1035] T 18a22320e5ba41c980e95c1a8d4cf99a P 1cd8517d70d94ae494e713df7eb61b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:16:17.993286 19041 consensus_queue.cc:1035] T 18a22320e5ba41c980e95c1a8d4cf99a P 1cd8517d70d94ae494e713df7eb61b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 220.467 ms
time per row: 0.110233 ms
I20240503 02:16:18.261575 17618 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44922:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:16:18.270237 17742 tablet_service.cc:1467] Processing CreateTablet for tablet 6315543b56a3474da4a13026ad498f84 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9d5524876b2a42ebbe1f4d61912135a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:18.270583 18005 tablet_service.cc:1467] Processing CreateTablet for tablet 6315543b56a3474da4a13026ad498f84 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9d5524876b2a42ebbe1f4d61912135a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:18.270726 17742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6315543b56a3474da4a13026ad498f84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:18.271034 18005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6315543b56a3474da4a13026ad498f84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:18.271035 17875 tablet_service.cc:1467] Processing CreateTablet for tablet 6315543b56a3474da4a13026ad498f84 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9d5524876b2a42ebbe1f4d61912135a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:18.271450 17875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6315543b56a3474da4a13026ad498f84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:18.272154 18003 tablet_service.cc:1467] Processing CreateTablet for tablet 13152897990e485c94569e37f1eb92d0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9d5524876b2a42ebbe1f4d61912135a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:18.272511 18003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13152897990e485c94569e37f1eb92d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:18.278911 18135 tablet_service.cc:1467] Processing CreateTablet for tablet 07ffe3a634c14bd5b0fce34355a5ee4f (DEFAULT_TABLE table=rebalance_test_table_4 [id=9d5524876b2a42ebbe1f4d61912135a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:18.278941 17743 tablet_service.cc:1467] Processing CreateTablet for tablet 07ffe3a634c14bd5b0fce34355a5ee4f (DEFAULT_TABLE table=rebalance_test_table_4 [id=9d5524876b2a42ebbe1f4d61912135a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:18.279471 17743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07ffe3a634c14bd5b0fce34355a5ee4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:18.279476 18135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07ffe3a634c14bd5b0fce34355a5ee4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:18.281558 18133 tablet_service.cc:1467] Processing CreateTablet for tablet 13152897990e485c94569e37f1eb92d0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9d5524876b2a42ebbe1f4d61912135a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:18.282028 18133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13152897990e485c94569e37f1eb92d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:18.282280 19105 tablet_bootstrap.cc:492] T 6315543b56a3474da4a13026ad498f84 P 0f3838ee7ad04e94aa65ac750dad0833: Bootstrap starting.
I20240503 02:16:18.284019 19105 tablet_bootstrap.cc:654] T 6315543b56a3474da4a13026ad498f84 P 0f3838ee7ad04e94aa65ac750dad0833: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:18.285544 17744 tablet_service.cc:1467] Processing CreateTablet for tablet 13152897990e485c94569e37f1eb92d0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9d5524876b2a42ebbe1f4d61912135a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:18.285797 19104 tablet_bootstrap.cc:492] T 6315543b56a3474da4a13026ad498f84 P 836b43a7ee9146948319b8d37434b630: Bootstrap starting.
I20240503 02:16:18.285960 17744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13152897990e485c94569e37f1eb92d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:18.288488 19104 tablet_bootstrap.cc:654] T 6315543b56a3474da4a13026ad498f84 P 836b43a7ee9146948319b8d37434b630: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:18.289975 17874 tablet_service.cc:1467] Processing CreateTablet for tablet 07ffe3a634c14bd5b0fce34355a5ee4f (DEFAULT_TABLE table=rebalance_test_table_4 [id=9d5524876b2a42ebbe1f4d61912135a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:18.290558 17874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07ffe3a634c14bd5b0fce34355a5ee4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:18.293177 19105 tablet_bootstrap.cc:492] T 6315543b56a3474da4a13026ad498f84 P 0f3838ee7ad04e94aa65ac750dad0833: No bootstrap required, opened a new log
I20240503 02:16:18.293336 19105 ts_tablet_manager.cc:1367] T 6315543b56a3474da4a13026ad498f84 P 0f3838ee7ad04e94aa65ac750dad0833: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.008s
I20240503 02:16:18.294219 19105 raft_consensus.cc:348] T 6315543b56a3474da4a13026ad498f84 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:18.294449 19105 raft_consensus.cc:374] T 6315543b56a3474da4a13026ad498f84 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:18.294559 19105 raft_consensus.cc:729] T 6315543b56a3474da4a13026ad498f84 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Initialized, Role: FOLLOWER
I20240503 02:16:18.294844 19105 consensus_queue.cc:260] T 6315543b56a3474da4a13026ad498f84 P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:18.295440 19105 ts_tablet_manager.cc:1398] T 6315543b56a3474da4a13026ad498f84 P 0f3838ee7ad04e94aa65ac750dad0833: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:16:18.299147 19105 tablet_bootstrap.cc:492] T 07ffe3a634c14bd5b0fce34355a5ee4f P 0f3838ee7ad04e94aa65ac750dad0833: Bootstrap starting.
I20240503 02:16:18.299196 19106 tablet_bootstrap.cc:492] T 6315543b56a3474da4a13026ad498f84 P 1cd8517d70d94ae494e713df7eb61b12: Bootstrap starting.
I20240503 02:16:18.300863 19105 tablet_bootstrap.cc:654] T 07ffe3a634c14bd5b0fce34355a5ee4f P 0f3838ee7ad04e94aa65ac750dad0833: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:18.300930 19106 tablet_bootstrap.cc:654] T 6315543b56a3474da4a13026ad498f84 P 1cd8517d70d94ae494e713df7eb61b12: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:18.309099 19105 tablet_bootstrap.cc:492] T 07ffe3a634c14bd5b0fce34355a5ee4f P 0f3838ee7ad04e94aa65ac750dad0833: No bootstrap required, opened a new log
I20240503 02:16:18.309201 19106 tablet_bootstrap.cc:492] T 6315543b56a3474da4a13026ad498f84 P 1cd8517d70d94ae494e713df7eb61b12: No bootstrap required, opened a new log
I20240503 02:16:18.309360 19105 ts_tablet_manager.cc:1367] T 07ffe3a634c14bd5b0fce34355a5ee4f P 0f3838ee7ad04e94aa65ac750dad0833: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240503 02:16:18.309384 19106 ts_tablet_manager.cc:1367] T 6315543b56a3474da4a13026ad498f84 P 1cd8517d70d94ae494e713df7eb61b12: Time spent bootstrapping tablet: real 0.010s user 0.010s sys 0.000s
I20240503 02:16:18.310240 19106 raft_consensus.cc:348] T 6315543b56a3474da4a13026ad498f84 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:18.310213 19105 raft_consensus.cc:348] T 07ffe3a634c14bd5b0fce34355a5ee4f P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:18.310447 19105 raft_consensus.cc:374] T 07ffe3a634c14bd5b0fce34355a5ee4f P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:18.310536 19106 raft_consensus.cc:374] T 6315543b56a3474da4a13026ad498f84 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:18.310561 19105 raft_consensus.cc:729] T 07ffe3a634c14bd5b0fce34355a5ee4f P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Initialized, Role: FOLLOWER
I20240503 02:16:18.310642 19106 raft_consensus.cc:729] T 6315543b56a3474da4a13026ad498f84 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Initialized, Role: FOLLOWER
I20240503 02:16:18.310942 19105 consensus_queue.cc:260] T 07ffe3a634c14bd5b0fce34355a5ee4f P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:18.311009 19106 consensus_queue.cc:260] T 6315543b56a3474da4a13026ad498f84 P 1cd8517d70d94ae494e713df7eb61b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:18.311654 19106 ts_tablet_manager.cc:1398] T 6315543b56a3474da4a13026ad498f84 P 1cd8517d70d94ae494e713df7eb61b12: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:16:18.312012 19106 tablet_bootstrap.cc:492] T 13152897990e485c94569e37f1eb92d0 P 1cd8517d70d94ae494e713df7eb61b12: Bootstrap starting.
I20240503 02:16:18.313969 19106 tablet_bootstrap.cc:654] T 13152897990e485c94569e37f1eb92d0 P 1cd8517d70d94ae494e713df7eb61b12: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:18.314424 19105 ts_tablet_manager.cc:1398] T 07ffe3a634c14bd5b0fce34355a5ee4f P 0f3838ee7ad04e94aa65ac750dad0833: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:16:18.319061 19106 tablet_bootstrap.cc:492] T 13152897990e485c94569e37f1eb92d0 P 1cd8517d70d94ae494e713df7eb61b12: No bootstrap required, opened a new log
I20240503 02:16:18.319226 19106 ts_tablet_manager.cc:1367] T 13152897990e485c94569e37f1eb92d0 P 1cd8517d70d94ae494e713df7eb61b12: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20240503 02:16:18.319599 19111 tablet_bootstrap.cc:492] T 07ffe3a634c14bd5b0fce34355a5ee4f P 4325d7d0ed484ad78a6ee54e9a003431: Bootstrap starting.
I20240503 02:16:18.319962 19104 tablet_bootstrap.cc:492] T 6315543b56a3474da4a13026ad498f84 P 836b43a7ee9146948319b8d37434b630: No bootstrap required, opened a new log
I20240503 02:16:18.320101 19104 ts_tablet_manager.cc:1367] T 6315543b56a3474da4a13026ad498f84 P 836b43a7ee9146948319b8d37434b630: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.016s
I20240503 02:16:18.319988 19106 raft_consensus.cc:348] T 13152897990e485c94569e37f1eb92d0 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:18.320190 19106 raft_consensus.cc:374] T 13152897990e485c94569e37f1eb92d0 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:18.320271 19106 raft_consensus.cc:729] T 13152897990e485c94569e37f1eb92d0 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Initialized, Role: FOLLOWER
I20240503 02:16:18.320514 19106 consensus_queue.cc:260] T 13152897990e485c94569e37f1eb92d0 P 1cd8517d70d94ae494e713df7eb61b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:18.320771 19104 raft_consensus.cc:348] T 6315543b56a3474da4a13026ad498f84 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:18.321065 19104 raft_consensus.cc:374] T 6315543b56a3474da4a13026ad498f84 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:18.321202 19106 ts_tablet_manager.cc:1398] T 13152897990e485c94569e37f1eb92d0 P 1cd8517d70d94ae494e713df7eb61b12: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 02:16:18.321210 19104 raft_consensus.cc:729] T 6315543b56a3474da4a13026ad498f84 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Initialized, Role: FOLLOWER
I20240503 02:16:18.321403 19111 tablet_bootstrap.cc:654] T 07ffe3a634c14bd5b0fce34355a5ee4f P 4325d7d0ed484ad78a6ee54e9a003431: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:18.321472 19104 consensus_queue.cc:260] T 6315543b56a3474da4a13026ad498f84 P 836b43a7ee9146948319b8d37434b630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:18.322163 19104 ts_tablet_manager.cc:1398] T 6315543b56a3474da4a13026ad498f84 P 836b43a7ee9146948319b8d37434b630: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:16:18.322845 19104 tablet_bootstrap.cc:492] T 13152897990e485c94569e37f1eb92d0 P 836b43a7ee9146948319b8d37434b630: Bootstrap starting.
I20240503 02:16:18.323621 19111 tablet_bootstrap.cc:492] T 07ffe3a634c14bd5b0fce34355a5ee4f P 4325d7d0ed484ad78a6ee54e9a003431: No bootstrap required, opened a new log
I20240503 02:16:18.323796 19111 ts_tablet_manager.cc:1367] T 07ffe3a634c14bd5b0fce34355a5ee4f P 4325d7d0ed484ad78a6ee54e9a003431: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 02:16:18.324587 19104 tablet_bootstrap.cc:654] T 13152897990e485c94569e37f1eb92d0 P 836b43a7ee9146948319b8d37434b630: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:18.325327 19111 raft_consensus.cc:348] T 07ffe3a634c14bd5b0fce34355a5ee4f P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:18.325544 19111 raft_consensus.cc:374] T 07ffe3a634c14bd5b0fce34355a5ee4f P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:18.325685 19111 raft_consensus.cc:729] T 07ffe3a634c14bd5b0fce34355a5ee4f P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4325d7d0ed484ad78a6ee54e9a003431, State: Initialized, Role: FOLLOWER
I20240503 02:16:18.325939 19111 consensus_queue.cc:260] T 07ffe3a634c14bd5b0fce34355a5ee4f P 4325d7d0ed484ad78a6ee54e9a003431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:18.326608 19111 ts_tablet_manager.cc:1398] T 07ffe3a634c14bd5b0fce34355a5ee4f P 4325d7d0ed484ad78a6ee54e9a003431: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:16:18.326925 19111 tablet_bootstrap.cc:492] T 13152897990e485c94569e37f1eb92d0 P 4325d7d0ed484ad78a6ee54e9a003431: Bootstrap starting.
I20240503 02:16:18.328244 19111 tablet_bootstrap.cc:654] T 13152897990e485c94569e37f1eb92d0 P 4325d7d0ed484ad78a6ee54e9a003431: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:18.332489 19104 tablet_bootstrap.cc:492] T 13152897990e485c94569e37f1eb92d0 P 836b43a7ee9146948319b8d37434b630: No bootstrap required, opened a new log
I20240503 02:16:18.332623 19104 ts_tablet_manager.cc:1367] T 13152897990e485c94569e37f1eb92d0 P 836b43a7ee9146948319b8d37434b630: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20240503 02:16:18.333240 19104 raft_consensus.cc:348] T 13152897990e485c94569e37f1eb92d0 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:18.333413 19104 raft_consensus.cc:374] T 13152897990e485c94569e37f1eb92d0 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:18.333485 19104 raft_consensus.cc:729] T 13152897990e485c94569e37f1eb92d0 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Initialized, Role: FOLLOWER
I20240503 02:16:18.333724 19104 consensus_queue.cc:260] T 13152897990e485c94569e37f1eb92d0 P 836b43a7ee9146948319b8d37434b630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:18.334262 19111 tablet_bootstrap.cc:492] T 13152897990e485c94569e37f1eb92d0 P 4325d7d0ed484ad78a6ee54e9a003431: No bootstrap required, opened a new log
I20240503 02:16:18.334407 19111 ts_tablet_manager.cc:1367] T 13152897990e485c94569e37f1eb92d0 P 4325d7d0ed484ad78a6ee54e9a003431: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240503 02:16:18.334990 19104 ts_tablet_manager.cc:1398] T 13152897990e485c94569e37f1eb92d0 P 836b43a7ee9146948319b8d37434b630: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:16:18.335299 19104 tablet_bootstrap.cc:492] T 07ffe3a634c14bd5b0fce34355a5ee4f P 836b43a7ee9146948319b8d37434b630: Bootstrap starting.
I20240503 02:16:18.335223 19111 raft_consensus.cc:348] T 13152897990e485c94569e37f1eb92d0 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:18.335393 19111 raft_consensus.cc:374] T 13152897990e485c94569e37f1eb92d0 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:18.335467 19111 raft_consensus.cc:729] T 13152897990e485c94569e37f1eb92d0 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4325d7d0ed484ad78a6ee54e9a003431, State: Initialized, Role: FOLLOWER
I20240503 02:16:18.335685 19111 consensus_queue.cc:260] T 13152897990e485c94569e37f1eb92d0 P 4325d7d0ed484ad78a6ee54e9a003431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:18.336200 19111 ts_tablet_manager.cc:1398] T 13152897990e485c94569e37f1eb92d0 P 4325d7d0ed484ad78a6ee54e9a003431: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:16:18.337528 19104 tablet_bootstrap.cc:654] T 07ffe3a634c14bd5b0fce34355a5ee4f P 836b43a7ee9146948319b8d37434b630: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:18.339221 19104 tablet_bootstrap.cc:492] T 07ffe3a634c14bd5b0fce34355a5ee4f P 836b43a7ee9146948319b8d37434b630: No bootstrap required, opened a new log
I20240503 02:16:18.339356 19104 ts_tablet_manager.cc:1367] T 07ffe3a634c14bd5b0fce34355a5ee4f P 836b43a7ee9146948319b8d37434b630: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:16:18.340116 19104 raft_consensus.cc:348] T 07ffe3a634c14bd5b0fce34355a5ee4f P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:18.340255 19104 raft_consensus.cc:374] T 07ffe3a634c14bd5b0fce34355a5ee4f P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:18.340317 19104 raft_consensus.cc:729] T 07ffe3a634c14bd5b0fce34355a5ee4f P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Initialized, Role: FOLLOWER
I20240503 02:16:18.340474 19104 consensus_queue.cc:260] T 07ffe3a634c14bd5b0fce34355a5ee4f P 836b43a7ee9146948319b8d37434b630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:18.340854 19104 ts_tablet_manager.cc:1398] T 07ffe3a634c14bd5b0fce34355a5ee4f P 836b43a7ee9146948319b8d37434b630: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:16:18.446475 18870 raft_consensus.cc:482] T 07ffe3a634c14bd5b0fce34355a5ee4f P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:18.446687 18870 raft_consensus.cc:504] T 07ffe3a634c14bd5b0fce34355a5ee4f P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:18.447412 18870 leader_election.cc:288] T 07ffe3a634c14bd5b0fce34355a5ee4f P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837), 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703)
I20240503 02:16:18.447898 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ffe3a634c14bd5b0fce34355a5ee4f" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431" is_pre_election: true
I20240503 02:16:18.447886 17764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ffe3a634c14bd5b0fce34355a5ee4f" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836b43a7ee9146948319b8d37434b630" is_pre_election: true
I20240503 02:16:18.448110 18155 raft_consensus.cc:2454] T 07ffe3a634c14bd5b0fce34355a5ee4f P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 0.
I20240503 02:16:18.448097 17764 raft_consensus.cc:2454] T 07ffe3a634c14bd5b0fce34355a5ee4f P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 0.
I20240503 02:16:18.448446 17828 leader_election.cc:302] T 07ffe3a634c14bd5b0fce34355a5ee4f P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:18.448699 19117 raft_consensus.cc:2781] T 07ffe3a634c14bd5b0fce34355a5ee4f P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:16:18.448796 19117 raft_consensus.cc:482] T 07ffe3a634c14bd5b0fce34355a5ee4f P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:16:18.448925 19117 raft_consensus.cc:3037] T 07ffe3a634c14bd5b0fce34355a5ee4f P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:18.449160 18826 raft_consensus.cc:482] T 6315543b56a3474da4a13026ad498f84 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:18.449322 18826 raft_consensus.cc:504] T 6315543b56a3474da4a13026ad498f84 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:18.449986 18826 leader_election.cc:288] T 6315543b56a3474da4a13026ad498f84 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647), 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759)
I20240503 02:16:18.450564 18025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6315543b56a3474da4a13026ad498f84" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd8517d70d94ae494e713df7eb61b12" is_pre_election: true
I20240503 02:16:18.450626 17894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6315543b56a3474da4a13026ad498f84" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833" is_pre_election: true
I20240503 02:16:18.450778 18025 raft_consensus.cc:2454] T 6315543b56a3474da4a13026ad498f84 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 0.
I20240503 02:16:18.450841 17894 raft_consensus.cc:2454] T 6315543b56a3474da4a13026ad498f84 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 0.
I20240503 02:16:18.451202 17700 leader_election.cc:302] T 6315543b56a3474da4a13026ad498f84 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cd8517d70d94ae494e713df7eb61b12, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:18.451473 18826 raft_consensus.cc:2781] T 6315543b56a3474da4a13026ad498f84 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:16:18.451563 18826 raft_consensus.cc:482] T 6315543b56a3474da4a13026ad498f84 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:16:18.451663 18826 raft_consensus.cc:3037] T 6315543b56a3474da4a13026ad498f84 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:18.451758 19117 raft_consensus.cc:504] T 07ffe3a634c14bd5b0fce34355a5ee4f P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:18.452292 19117 leader_election.cc:288] T 07ffe3a634c14bd5b0fce34355a5ee4f P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 1 election: Requested vote from peers 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837), 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703)
I20240503 02:16:18.452675 17764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ffe3a634c14bd5b0fce34355a5ee4f" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836b43a7ee9146948319b8d37434b630"
I20240503 02:16:18.452703 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ffe3a634c14bd5b0fce34355a5ee4f" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431"
I20240503 02:16:18.452849 17764 raft_consensus.cc:3037] T 07ffe3a634c14bd5b0fce34355a5ee4f P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:18.452932 18155 raft_consensus.cc:3037] T 07ffe3a634c14bd5b0fce34355a5ee4f P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:18.454365 18826 raft_consensus.cc:504] T 6315543b56a3474da4a13026ad498f84 P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:18.454774 17764 raft_consensus.cc:2454] T 07ffe3a634c14bd5b0fce34355a5ee4f P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 1.
I20240503 02:16:18.455324 17894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6315543b56a3474da4a13026ad498f84" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
I20240503 02:16:18.455513 17894 raft_consensus.cc:3037] T 6315543b56a3474da4a13026ad498f84 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:18.455744 18025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6315543b56a3474da4a13026ad498f84" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:18.455946 18025 raft_consensus.cc:3037] T 6315543b56a3474da4a13026ad498f84 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:18.457459 17894 raft_consensus.cc:2454] T 6315543b56a3474da4a13026ad498f84 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 1.
I20240503 02:16:18.457980 17697 leader_election.cc:302] T 6315543b56a3474da4a13026ad498f84 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:18.458230 19097 raft_consensus.cc:2781] T 6315543b56a3474da4a13026ad498f84 P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:16:18.458393 19097 raft_consensus.cc:686] T 6315543b56a3474da4a13026ad498f84 P 836b43a7ee9146948319b8d37434b630 [term 1 LEADER]: Becoming Leader. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Running, Role: LEADER
I20240503 02:16:18.458890 18025 raft_consensus.cc:2454] T 6315543b56a3474da4a13026ad498f84 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 1.
I20240503 02:16:18.458777 19097 consensus_queue.cc:237] T 6315543b56a3474da4a13026ad498f84 P 836b43a7ee9146948319b8d37434b630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:18.459383 18826 leader_election.cc:288] T 6315543b56a3474da4a13026ad498f84 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 1 election: Requested vote from peers 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647), 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759)
I20240503 02:16:18.461313 17828 leader_election.cc:302] T 07ffe3a634c14bd5b0fce34355a5ee4f P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:18.461895 17619 catalog_manager.cc:5484] T 6315543b56a3474da4a13026ad498f84 P 836b43a7ee9146948319b8d37434b630 reported cstate change: term changed from 0 to 1, leader changed from <none> to 836b43a7ee9146948319b8d37434b630 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "836b43a7ee9146948319b8d37434b630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } }
I20240503 02:16:18.462652 18155 raft_consensus.cc:2454] T 07ffe3a634c14bd5b0fce34355a5ee4f P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 1.
I20240503 02:16:18.463201 19117 raft_consensus.cc:2781] T 07ffe3a634c14bd5b0fce34355a5ee4f P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:16:18.463367 19117 raft_consensus.cc:686] T 07ffe3a634c14bd5b0fce34355a5ee4f P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 LEADER]: Becoming Leader. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:18.463567 19117 consensus_queue.cc:237] T 07ffe3a634c14bd5b0fce34355a5ee4f P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:18.467260 17619 catalog_manager.cc:5484] T 07ffe3a634c14bd5b0fce34355a5ee4f P 0f3838ee7ad04e94aa65ac750dad0833 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2). New cstate: current_term: 1 leader_uuid: "0f3838ee7ad04e94aa65ac750dad0833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } }
I20240503 02:16:18.597152 19097 raft_consensus.cc:482] T 13152897990e485c94569e37f1eb92d0 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:18.597390 19097 raft_consensus.cc:504] T 13152897990e485c94569e37f1eb92d0 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:18.597909 19097 leader_election.cc:288] T 13152897990e485c94569e37f1eb92d0 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703), 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759)
I20240503 02:16:18.598373 18025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13152897990e485c94569e37f1eb92d0" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd8517d70d94ae494e713df7eb61b12" is_pre_election: true
I20240503 02:16:18.598381 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13152897990e485c94569e37f1eb92d0" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431" is_pre_election: true
I20240503 02:16:18.598604 18155 raft_consensus.cc:2454] T 13152897990e485c94569e37f1eb92d0 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 0.
I20240503 02:16:18.598598 18025 raft_consensus.cc:2454] T 13152897990e485c94569e37f1eb92d0 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 0.
I20240503 02:16:18.598968 17700 leader_election.cc:302] T 13152897990e485c94569e37f1eb92d0 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cd8517d70d94ae494e713df7eb61b12, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:18.599262 19097 raft_consensus.cc:2781] T 13152897990e485c94569e37f1eb92d0 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:16:18.599349 19097 raft_consensus.cc:482] T 13152897990e485c94569e37f1eb92d0 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:16:18.599455 19097 raft_consensus.cc:3037] T 13152897990e485c94569e37f1eb92d0 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:18.601248 19097 raft_consensus.cc:504] T 13152897990e485c94569e37f1eb92d0 P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:18.601764 19097 leader_election.cc:288] T 13152897990e485c94569e37f1eb92d0 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 1 election: Requested vote from peers 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703), 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759)
I20240503 02:16:18.602161 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13152897990e485c94569e37f1eb92d0" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431"
I20240503 02:16:18.602191 18025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13152897990e485c94569e37f1eb92d0" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:18.602326 18155 raft_consensus.cc:3037] T 13152897990e485c94569e37f1eb92d0 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:18.602357 18025 raft_consensus.cc:3037] T 13152897990e485c94569e37f1eb92d0 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:18.604175 18025 raft_consensus.cc:2454] T 13152897990e485c94569e37f1eb92d0 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 1.
I20240503 02:16:18.604187 18155 raft_consensus.cc:2454] T 13152897990e485c94569e37f1eb92d0 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 1.
I20240503 02:16:18.604507 17697 leader_election.cc:302] T 13152897990e485c94569e37f1eb92d0 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4325d7d0ed484ad78a6ee54e9a003431, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:18.604746 19097 raft_consensus.cc:2781] T 13152897990e485c94569e37f1eb92d0 P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:16:18.604918 19097 raft_consensus.cc:686] T 13152897990e485c94569e37f1eb92d0 P 836b43a7ee9146948319b8d37434b630 [term 1 LEADER]: Becoming Leader. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Running, Role: LEADER
I20240503 02:16:18.605180 19097 consensus_queue.cc:237] T 13152897990e485c94569e37f1eb92d0 P 836b43a7ee9146948319b8d37434b630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:18.607789 17619 catalog_manager.cc:5484] T 13152897990e485c94569e37f1eb92d0 P 836b43a7ee9146948319b8d37434b630 reported cstate change: term changed from 0 to 1, leader changed from <none> to 836b43a7ee9146948319b8d37434b630 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "836b43a7ee9146948319b8d37434b630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } }
I20240503 02:16:18.863602 19097 consensus_queue.cc:1035] T 6315543b56a3474da4a13026ad498f84 P 836b43a7ee9146948319b8d37434b630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:16:18.869511 19097 consensus_queue.cc:1035] T 6315543b56a3474da4a13026ad498f84 P 836b43a7ee9146948319b8d37434b630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:16:18.886148 19117 consensus_queue.cc:1035] T 07ffe3a634c14bd5b0fce34355a5ee4f P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:16:18.891765 19117 consensus_queue.cc:1035] T 07ffe3a634c14bd5b0fce34355a5ee4f P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:16:19.015157 19121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:16:19.015475 19121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:16:19.023573 19121 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 02:16:19.139662 19097 consensus_queue.cc:1035] T 13152897990e485c94569e37f1eb92d0 P 836b43a7ee9146948319b8d37434b630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:16:19.143962 19097 consensus_queue.cc:1035] T 13152897990e485c94569e37f1eb92d0 P 836b43a7ee9146948319b8d37434b630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:16:19.292428 19162 mvcc.cc:205] Tried to move back new op lower bound from 7023421764340051968 to 7023421761968553984. Current Snapshot: MvccSnapshot[applied={T|T < 7023421764303458304}]
INSERT report
rows total: 2000
time total: 255.296 ms
time per row: 0.127648 ms
I20240503 02:16:19.435504 17623 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44922:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:16:19.444278 17874 tablet_service.cc:1467] Processing CreateTablet for tablet 0f8c3cd667594af4969548ba820e0164 (DEFAULT_TABLE table=rebalance_test_table_5 [id=bcf248c0f9694fd29b38fd981d808851]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:19.444249 18135 tablet_service.cc:1467] Processing CreateTablet for tablet 0f8c3cd667594af4969548ba820e0164 (DEFAULT_TABLE table=rebalance_test_table_5 [id=bcf248c0f9694fd29b38fd981d808851]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:19.444757 17874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f8c3cd667594af4969548ba820e0164. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:19.444769 18135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f8c3cd667594af4969548ba820e0164. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:19.445257 18133 tablet_service.cc:1467] Processing CreateTablet for tablet 0122399369d14e57a55a1415c710e72f (DEFAULT_TABLE table=rebalance_test_table_5 [id=bcf248c0f9694fd29b38fd981d808851]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:19.445375 17744 tablet_service.cc:1467] Processing CreateTablet for tablet 6bc5c6531d5144bf80f1b388f23703e9 (DEFAULT_TABLE table=rebalance_test_table_5 [id=bcf248c0f9694fd29b38fd981d808851]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:19.445611 18133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0122399369d14e57a55a1415c710e72f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:19.445777 17744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bc5c6531d5144bf80f1b388f23703e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:19.445749 18003 tablet_service.cc:1467] Processing CreateTablet for tablet 0f8c3cd667594af4969548ba820e0164 (DEFAULT_TABLE table=rebalance_test_table_5 [id=bcf248c0f9694fd29b38fd981d808851]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:19.446144 18003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f8c3cd667594af4969548ba820e0164. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:19.450096 18005 tablet_service.cc:1467] Processing CreateTablet for tablet 0122399369d14e57a55a1415c710e72f (DEFAULT_TABLE table=rebalance_test_table_5 [id=bcf248c0f9694fd29b38fd981d808851]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:19.450515 18005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0122399369d14e57a55a1415c710e72f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:19.454136 17875 tablet_service.cc:1467] Processing CreateTablet for tablet 0122399369d14e57a55a1415c710e72f (DEFAULT_TABLE table=rebalance_test_table_5 [id=bcf248c0f9694fd29b38fd981d808851]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:19.454522 17875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0122399369d14e57a55a1415c710e72f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:19.454646 18004 tablet_service.cc:1467] Processing CreateTablet for tablet 6bc5c6531d5144bf80f1b388f23703e9 (DEFAULT_TABLE table=rebalance_test_table_5 [id=bcf248c0f9694fd29b38fd981d808851]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:19.455144 18004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bc5c6531d5144bf80f1b388f23703e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:19.457376 17873 tablet_service.cc:1467] Processing CreateTablet for tablet 6bc5c6531d5144bf80f1b388f23703e9 (DEFAULT_TABLE table=rebalance_test_table_5 [id=bcf248c0f9694fd29b38fd981d808851]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:19.457728 17873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bc5c6531d5144bf80f1b388f23703e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:19.473637 19191 tablet_bootstrap.cc:492] T 0f8c3cd667594af4969548ba820e0164 P 1cd8517d70d94ae494e713df7eb61b12: Bootstrap starting.
I20240503 02:16:19.474237 19190 tablet_bootstrap.cc:492] T 0f8c3cd667594af4969548ba820e0164 P 0f3838ee7ad04e94aa65ac750dad0833: Bootstrap starting.
I20240503 02:16:19.475824 19191 tablet_bootstrap.cc:654] T 0f8c3cd667594af4969548ba820e0164 P 1cd8517d70d94ae494e713df7eb61b12: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:19.478754 19190 tablet_bootstrap.cc:654] T 0f8c3cd667594af4969548ba820e0164 P 0f3838ee7ad04e94aa65ac750dad0833: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:19.481052 19191 tablet_bootstrap.cc:492] T 0f8c3cd667594af4969548ba820e0164 P 1cd8517d70d94ae494e713df7eb61b12: No bootstrap required, opened a new log
I20240503 02:16:19.481287 19191 ts_tablet_manager.cc:1367] T 0f8c3cd667594af4969548ba820e0164 P 1cd8517d70d94ae494e713df7eb61b12: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240503 02:16:19.482170 19191 raft_consensus.cc:348] T 0f8c3cd667594af4969548ba820e0164 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:19.482489 19191 raft_consensus.cc:374] T 0f8c3cd667594af4969548ba820e0164 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:19.482636 19191 raft_consensus.cc:729] T 0f8c3cd667594af4969548ba820e0164 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Initialized, Role: FOLLOWER
I20240503 02:16:19.483057 19191 consensus_queue.cc:260] T 0f8c3cd667594af4969548ba820e0164 P 1cd8517d70d94ae494e713df7eb61b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:19.484084 19191 ts_tablet_manager.cc:1398] T 0f8c3cd667594af4969548ba820e0164 P 1cd8517d70d94ae494e713df7eb61b12: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:16:19.484323 19189 tablet_bootstrap.cc:492] T 6bc5c6531d5144bf80f1b388f23703e9 P 836b43a7ee9146948319b8d37434b630: Bootstrap starting.
I20240503 02:16:19.486805 19191 tablet_bootstrap.cc:492] T 0122399369d14e57a55a1415c710e72f P 1cd8517d70d94ae494e713df7eb61b12: Bootstrap starting.
I20240503 02:16:19.486814 19189 tablet_bootstrap.cc:654] T 6bc5c6531d5144bf80f1b388f23703e9 P 836b43a7ee9146948319b8d37434b630: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:19.488991 19191 tablet_bootstrap.cc:654] T 0122399369d14e57a55a1415c710e72f P 1cd8517d70d94ae494e713df7eb61b12: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:19.489802 19188 tablet_bootstrap.cc:492] T 0f8c3cd667594af4969548ba820e0164 P 4325d7d0ed484ad78a6ee54e9a003431: Bootstrap starting.
I20240503 02:16:19.490700 19190 tablet_bootstrap.cc:492] T 0f8c3cd667594af4969548ba820e0164 P 0f3838ee7ad04e94aa65ac750dad0833: No bootstrap required, opened a new log
I20240503 02:16:19.490868 19190 ts_tablet_manager.cc:1367] T 0f8c3cd667594af4969548ba820e0164 P 0f3838ee7ad04e94aa65ac750dad0833: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240503 02:16:19.491657 19190 raft_consensus.cc:348] T 0f8c3cd667594af4969548ba820e0164 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:19.491873 19190 raft_consensus.cc:374] T 0f8c3cd667594af4969548ba820e0164 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:19.491953 19190 raft_consensus.cc:729] T 0f8c3cd667594af4969548ba820e0164 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Initialized, Role: FOLLOWER
I20240503 02:16:19.492158 19190 consensus_queue.cc:260] T 0f8c3cd667594af4969548ba820e0164 P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:19.492719 19190 ts_tablet_manager.cc:1398] T 0f8c3cd667594af4969548ba820e0164 P 0f3838ee7ad04e94aa65ac750dad0833: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:16:19.493019 19190 tablet_bootstrap.cc:492] T 0122399369d14e57a55a1415c710e72f P 0f3838ee7ad04e94aa65ac750dad0833: Bootstrap starting.
I20240503 02:16:19.494609 19190 tablet_bootstrap.cc:654] T 0122399369d14e57a55a1415c710e72f P 0f3838ee7ad04e94aa65ac750dad0833: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:19.500332 19188 tablet_bootstrap.cc:654] T 0f8c3cd667594af4969548ba820e0164 P 4325d7d0ed484ad78a6ee54e9a003431: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:19.506011 19191 tablet_bootstrap.cc:492] T 0122399369d14e57a55a1415c710e72f P 1cd8517d70d94ae494e713df7eb61b12: No bootstrap required, opened a new log
I20240503 02:16:19.506098 19190 tablet_bootstrap.cc:492] T 0122399369d14e57a55a1415c710e72f P 0f3838ee7ad04e94aa65ac750dad0833: No bootstrap required, opened a new log
I20240503 02:16:19.506260 19190 ts_tablet_manager.cc:1367] T 0122399369d14e57a55a1415c710e72f P 0f3838ee7ad04e94aa65ac750dad0833: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20240503 02:16:19.506304 19191 ts_tablet_manager.cc:1367] T 0122399369d14e57a55a1415c710e72f P 1cd8517d70d94ae494e713df7eb61b12: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.007s
I20240503 02:16:19.506712 19188 tablet_bootstrap.cc:492] T 0f8c3cd667594af4969548ba820e0164 P 4325d7d0ed484ad78a6ee54e9a003431: No bootstrap required, opened a new log
I20240503 02:16:19.506870 19188 ts_tablet_manager.cc:1367] T 0f8c3cd667594af4969548ba820e0164 P 4325d7d0ed484ad78a6ee54e9a003431: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240503 02:16:19.506991 19190 raft_consensus.cc:348] T 0122399369d14e57a55a1415c710e72f P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:19.507225 19190 raft_consensus.cc:374] T 0122399369d14e57a55a1415c710e72f P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:19.507292 19190 raft_consensus.cc:729] T 0122399369d14e57a55a1415c710e72f P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Initialized, Role: FOLLOWER
I20240503 02:16:19.507544 19190 consensus_queue.cc:260] T 0122399369d14e57a55a1415c710e72f P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:19.507623 19188 raft_consensus.cc:348] T 0f8c3cd667594af4969548ba820e0164 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:19.507830 19188 raft_consensus.cc:374] T 0f8c3cd667594af4969548ba820e0164 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:19.507598 19191 raft_consensus.cc:348] T 0122399369d14e57a55a1415c710e72f P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:19.507925 19188 raft_consensus.cc:729] T 0f8c3cd667594af4969548ba820e0164 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4325d7d0ed484ad78a6ee54e9a003431, State: Initialized, Role: FOLLOWER
I20240503 02:16:19.508071 19191 raft_consensus.cc:374] T 0122399369d14e57a55a1415c710e72f P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:19.508212 19191 raft_consensus.cc:729] T 0122399369d14e57a55a1415c710e72f P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Initialized, Role: FOLLOWER
I20240503 02:16:19.508777 19190 ts_tablet_manager.cc:1398] T 0122399369d14e57a55a1415c710e72f P 0f3838ee7ad04e94aa65ac750dad0833: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:16:19.509081 19190 tablet_bootstrap.cc:492] T 6bc5c6531d5144bf80f1b388f23703e9 P 0f3838ee7ad04e94aa65ac750dad0833: Bootstrap starting.
I20240503 02:16:19.509570 19188 consensus_queue.cc:260] T 0f8c3cd667594af4969548ba820e0164 P 4325d7d0ed484ad78a6ee54e9a003431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:19.510334 19188 ts_tablet_manager.cc:1398] T 0f8c3cd667594af4969548ba820e0164 P 4325d7d0ed484ad78a6ee54e9a003431: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:16:19.510763 19188 tablet_bootstrap.cc:492] T 0122399369d14e57a55a1415c710e72f P 4325d7d0ed484ad78a6ee54e9a003431: Bootstrap starting.
I20240503 02:16:19.511271 19190 tablet_bootstrap.cc:654] T 6bc5c6531d5144bf80f1b388f23703e9 P 0f3838ee7ad04e94aa65ac750dad0833: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:19.512296 19188 tablet_bootstrap.cc:654] T 0122399369d14e57a55a1415c710e72f P 4325d7d0ed484ad78a6ee54e9a003431: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:19.514006 19189 tablet_bootstrap.cc:492] T 6bc5c6531d5144bf80f1b388f23703e9 P 836b43a7ee9146948319b8d37434b630: No bootstrap required, opened a new log
I20240503 02:16:19.514206 19189 ts_tablet_manager.cc:1367] T 6bc5c6531d5144bf80f1b388f23703e9 P 836b43a7ee9146948319b8d37434b630: Time spent bootstrapping tablet: real 0.030s user 0.014s sys 0.005s
I20240503 02:16:19.514608 19191 consensus_queue.cc:260] T 0122399369d14e57a55a1415c710e72f P 1cd8517d70d94ae494e713df7eb61b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:19.515141 19189 raft_consensus.cc:348] T 6bc5c6531d5144bf80f1b388f23703e9 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:19.515393 19189 raft_consensus.cc:374] T 6bc5c6531d5144bf80f1b388f23703e9 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:19.515558 19189 raft_consensus.cc:729] T 6bc5c6531d5144bf80f1b388f23703e9 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Initialized, Role: FOLLOWER
I20240503 02:16:19.515817 19189 consensus_queue.cc:260] T 6bc5c6531d5144bf80f1b388f23703e9 P 836b43a7ee9146948319b8d37434b630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:19.516516 19189 ts_tablet_manager.cc:1398] T 6bc5c6531d5144bf80f1b388f23703e9 P 836b43a7ee9146948319b8d37434b630: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:16:19.524796 19190 tablet_bootstrap.cc:492] T 6bc5c6531d5144bf80f1b388f23703e9 P 0f3838ee7ad04e94aa65ac750dad0833: No bootstrap required, opened a new log
I20240503 02:16:19.524945 19190 ts_tablet_manager.cc:1367] T 6bc5c6531d5144bf80f1b388f23703e9 P 0f3838ee7ad04e94aa65ac750dad0833: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.011s
I20240503 02:16:19.525704 19190 raft_consensus.cc:348] T 6bc5c6531d5144bf80f1b388f23703e9 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:19.525903 19190 raft_consensus.cc:374] T 6bc5c6531d5144bf80f1b388f23703e9 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:19.525985 19190 raft_consensus.cc:729] T 6bc5c6531d5144bf80f1b388f23703e9 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Initialized, Role: FOLLOWER
I20240503 02:16:19.526196 19190 consensus_queue.cc:260] T 6bc5c6531d5144bf80f1b388f23703e9 P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:19.526671 19190 ts_tablet_manager.cc:1398] T 6bc5c6531d5144bf80f1b388f23703e9 P 0f3838ee7ad04e94aa65ac750dad0833: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:16:19.528142 19191 ts_tablet_manager.cc:1398] T 0122399369d14e57a55a1415c710e72f P 1cd8517d70d94ae494e713df7eb61b12: Time spent starting tablet: real 0.022s user 0.000s sys 0.002s
I20240503 02:16:19.529129 19191 tablet_bootstrap.cc:492] T 6bc5c6531d5144bf80f1b388f23703e9 P 1cd8517d70d94ae494e713df7eb61b12: Bootstrap starting.
I20240503 02:16:19.530418 19188 tablet_bootstrap.cc:492] T 0122399369d14e57a55a1415c710e72f P 4325d7d0ed484ad78a6ee54e9a003431: No bootstrap required, opened a new log
I20240503 02:16:19.530573 19191 tablet_bootstrap.cc:654] T 6bc5c6531d5144bf80f1b388f23703e9 P 1cd8517d70d94ae494e713df7eb61b12: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:19.530599 19188 ts_tablet_manager.cc:1367] T 0122399369d14e57a55a1415c710e72f P 4325d7d0ed484ad78a6ee54e9a003431: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.000s
I20240503 02:16:19.531337 19188 raft_consensus.cc:348] T 0122399369d14e57a55a1415c710e72f P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:19.531549 19188 raft_consensus.cc:374] T 0122399369d14e57a55a1415c710e72f P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:19.531633 19188 raft_consensus.cc:729] T 0122399369d14e57a55a1415c710e72f P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4325d7d0ed484ad78a6ee54e9a003431, State: Initialized, Role: FOLLOWER
I20240503 02:16:19.531884 19188 consensus_queue.cc:260] T 0122399369d14e57a55a1415c710e72f P 4325d7d0ed484ad78a6ee54e9a003431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:19.532590 19188 ts_tablet_manager.cc:1398] T 0122399369d14e57a55a1415c710e72f P 4325d7d0ed484ad78a6ee54e9a003431: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:16:19.533293 19191 tablet_bootstrap.cc:492] T 6bc5c6531d5144bf80f1b388f23703e9 P 1cd8517d70d94ae494e713df7eb61b12: No bootstrap required, opened a new log
I20240503 02:16:19.533414 19191 ts_tablet_manager.cc:1367] T 6bc5c6531d5144bf80f1b388f23703e9 P 1cd8517d70d94ae494e713df7eb61b12: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:16:19.533987 19191 raft_consensus.cc:348] T 6bc5c6531d5144bf80f1b388f23703e9 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:19.534183 19191 raft_consensus.cc:374] T 6bc5c6531d5144bf80f1b388f23703e9 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:19.534301 19191 raft_consensus.cc:729] T 6bc5c6531d5144bf80f1b388f23703e9 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Initialized, Role: FOLLOWER
I20240503 02:16:19.534554 19191 consensus_queue.cc:260] T 6bc5c6531d5144bf80f1b388f23703e9 P 1cd8517d70d94ae494e713df7eb61b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:19.535121 19191 ts_tablet_manager.cc:1398] T 6bc5c6531d5144bf80f1b388f23703e9 P 1cd8517d70d94ae494e713df7eb61b12: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:16:19.592981 19163 raft_consensus.cc:482] T 0122399369d14e57a55a1415c710e72f P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:19.593197 19163 raft_consensus.cc:504] T 0122399369d14e57a55a1415c710e72f P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:19.593849 19163 leader_election.cc:288] T 0122399369d14e57a55a1415c710e72f P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647), 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703)
I20240503 02:16:19.594120 19163 raft_consensus.cc:482] T 0f8c3cd667594af4969548ba820e0164 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:19.594275 19163 raft_consensus.cc:504] T 0f8c3cd667594af4969548ba820e0164 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:19.594385 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0122399369d14e57a55a1415c710e72f" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431" is_pre_election: true
I20240503 02:16:19.594385 17894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0122399369d14e57a55a1415c710e72f" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833" is_pre_election: true
I20240503 02:16:19.594635 17894 raft_consensus.cc:2454] T 0122399369d14e57a55a1415c710e72f P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cd8517d70d94ae494e713df7eb61b12 in term 0.
I20240503 02:16:19.594635 18155 raft_consensus.cc:2454] T 0122399369d14e57a55a1415c710e72f P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cd8517d70d94ae494e713df7eb61b12 in term 0.
I20240503 02:16:19.594805 19163 leader_election.cc:288] T 0f8c3cd667594af4969548ba820e0164 P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647), 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703)
I20240503 02:16:19.595029 17958 leader_election.cc:302] T 0122399369d14e57a55a1415c710e72f P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 1cd8517d70d94ae494e713df7eb61b12; no voters:
I20240503 02:16:19.595239 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f8c3cd667594af4969548ba820e0164" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431" is_pre_election: true
I20240503 02:16:19.595321 19163 raft_consensus.cc:2781] T 0122399369d14e57a55a1415c710e72f P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:16:19.595273 17894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f8c3cd667594af4969548ba820e0164" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833" is_pre_election: true
I20240503 02:16:19.595404 19163 raft_consensus.cc:482] T 0122399369d14e57a55a1415c710e72f P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:16:19.595435 18155 raft_consensus.cc:2454] T 0f8c3cd667594af4969548ba820e0164 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cd8517d70d94ae494e713df7eb61b12 in term 0.
I20240503 02:16:19.595474 17894 raft_consensus.cc:2454] T 0f8c3cd667594af4969548ba820e0164 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cd8517d70d94ae494e713df7eb61b12 in term 0.
I20240503 02:16:19.595506 19163 raft_consensus.cc:3037] T 0122399369d14e57a55a1415c710e72f P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:19.595817 17958 leader_election.cc:302] T 0f8c3cd667594af4969548ba820e0164 P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 1cd8517d70d94ae494e713df7eb61b12; no voters:
I20240503 02:16:19.596112 18880 raft_consensus.cc:2781] T 0f8c3cd667594af4969548ba820e0164 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:16:19.596223 18880 raft_consensus.cc:482] T 0f8c3cd667594af4969548ba820e0164 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:16:19.596343 18880 raft_consensus.cc:3037] T 0f8c3cd667594af4969548ba820e0164 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:19.597676 19163 raft_consensus.cc:504] T 0122399369d14e57a55a1415c710e72f P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:19.598176 19163 leader_election.cc:288] T 0122399369d14e57a55a1415c710e72f P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 1 election: Requested vote from peers 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647), 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703)
I20240503 02:16:19.598532 17894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0122399369d14e57a55a1415c710e72f" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
I20240503 02:16:19.598606 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0122399369d14e57a55a1415c710e72f" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431"
I20240503 02:16:19.598693 17894 raft_consensus.cc:3037] T 0122399369d14e57a55a1415c710e72f P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:19.598806 18155 raft_consensus.cc:3037] T 0122399369d14e57a55a1415c710e72f P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:19.600528 17894 raft_consensus.cc:2454] T 0122399369d14e57a55a1415c710e72f P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cd8517d70d94ae494e713df7eb61b12 in term 1.
I20240503 02:16:19.600740 18155 raft_consensus.cc:2454] T 0122399369d14e57a55a1415c710e72f P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cd8517d70d94ae494e713df7eb61b12 in term 1.
I20240503 02:16:19.600894 17958 leader_election.cc:302] T 0122399369d14e57a55a1415c710e72f P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 1cd8517d70d94ae494e713df7eb61b12; no voters:
I20240503 02:16:19.601171 19163 raft_consensus.cc:2781] T 0122399369d14e57a55a1415c710e72f P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:16:19.601595 19163 raft_consensus.cc:686] T 0122399369d14e57a55a1415c710e72f P 1cd8517d70d94ae494e713df7eb61b12 [term 1 LEADER]: Becoming Leader. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Running, Role: LEADER
I20240503 02:16:19.601857 19163 consensus_queue.cc:237] T 0122399369d14e57a55a1415c710e72f P 1cd8517d70d94ae494e713df7eb61b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:19.604542 17623 catalog_manager.cc:5484] T 0122399369d14e57a55a1415c710e72f P 1cd8517d70d94ae494e713df7eb61b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3). New cstate: current_term: 1 leader_uuid: "1cd8517d70d94ae494e713df7eb61b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } }
I20240503 02:16:19.612788 19168 raft_consensus.cc:482] T 6bc5c6531d5144bf80f1b388f23703e9 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:19.612946 19168 raft_consensus.cc:504] T 6bc5c6531d5144bf80f1b388f23703e9 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:19.613804 19168 leader_election.cc:288] T 6bc5c6531d5144bf80f1b388f23703e9 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759), 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647)
I20240503 02:16:19.614151 18025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bc5c6531d5144bf80f1b388f23703e9" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd8517d70d94ae494e713df7eb61b12" is_pre_election: true
I20240503 02:16:19.614401 18025 raft_consensus.cc:2454] T 6bc5c6531d5144bf80f1b388f23703e9 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 0.
I20240503 02:16:19.614389 17894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bc5c6531d5144bf80f1b388f23703e9" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833" is_pre_election: true
I20240503 02:16:19.614593 17894 raft_consensus.cc:2454] T 6bc5c6531d5144bf80f1b388f23703e9 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 0.
I20240503 02:16:19.614878 17700 leader_election.cc:302] T 6bc5c6531d5144bf80f1b388f23703e9 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cd8517d70d94ae494e713df7eb61b12, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:19.615105 19168 raft_consensus.cc:2781] T 6bc5c6531d5144bf80f1b388f23703e9 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:16:19.615217 19168 raft_consensus.cc:482] T 6bc5c6531d5144bf80f1b388f23703e9 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:16:19.615307 19168 raft_consensus.cc:3037] T 6bc5c6531d5144bf80f1b388f23703e9 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:19.615608 18880 raft_consensus.cc:504] T 0f8c3cd667594af4969548ba820e0164 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:19.616218 18880 leader_election.cc:288] T 0f8c3cd667594af4969548ba820e0164 P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 1 election: Requested vote from peers 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647), 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703)
I20240503 02:16:19.616596 17894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f8c3cd667594af4969548ba820e0164" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
I20240503 02:16:19.616753 17894 raft_consensus.cc:3037] T 0f8c3cd667594af4969548ba820e0164 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:19.617053 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f8c3cd667594af4969548ba820e0164" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431"
I20240503 02:16:19.617259 18155 raft_consensus.cc:3037] T 0f8c3cd667594af4969548ba820e0164 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:19.617594 19168 raft_consensus.cc:504] T 6bc5c6531d5144bf80f1b388f23703e9 P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:19.618317 19168 leader_election.cc:288] T 6bc5c6531d5144bf80f1b388f23703e9 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 1 election: Requested vote from peers 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759), 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647)
I20240503 02:16:19.618876 18025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bc5c6531d5144bf80f1b388f23703e9" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:19.618906 17895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bc5c6531d5144bf80f1b388f23703e9" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
I20240503 02:16:19.619088 17895 raft_consensus.cc:3037] T 6bc5c6531d5144bf80f1b388f23703e9 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:19.619102 18025 raft_consensus.cc:3037] T 6bc5c6531d5144bf80f1b388f23703e9 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:19.619163 17894 raft_consensus.cc:2454] T 0f8c3cd667594af4969548ba820e0164 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cd8517d70d94ae494e713df7eb61b12 in term 1.
I20240503 02:16:19.619655 17958 leader_election.cc:302] T 0f8c3cd667594af4969548ba820e0164 P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 1cd8517d70d94ae494e713df7eb61b12; no voters:
I20240503 02:16:19.619822 18155 raft_consensus.cc:2454] T 0f8c3cd667594af4969548ba820e0164 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cd8517d70d94ae494e713df7eb61b12 in term 1.
I20240503 02:16:19.619969 18880 raft_consensus.cc:2781] T 0f8c3cd667594af4969548ba820e0164 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:16:19.620124 18880 raft_consensus.cc:686] T 0f8c3cd667594af4969548ba820e0164 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 LEADER]: Becoming Leader. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Running, Role: LEADER
I20240503 02:16:19.620388 18880 consensus_queue.cc:237] T 0f8c3cd667594af4969548ba820e0164 P 1cd8517d70d94ae494e713df7eb61b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:19.621399 17895 raft_consensus.cc:2454] T 6bc5c6531d5144bf80f1b388f23703e9 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 1.
I20240503 02:16:19.621829 17697 leader_election.cc:302] T 6bc5c6531d5144bf80f1b388f23703e9 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:19.621912 18025 raft_consensus.cc:2454] T 6bc5c6531d5144bf80f1b388f23703e9 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 1.
I20240503 02:16:19.622151 19168 raft_consensus.cc:2781] T 6bc5c6531d5144bf80f1b388f23703e9 P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:16:19.622413 19168 raft_consensus.cc:686] T 6bc5c6531d5144bf80f1b388f23703e9 P 836b43a7ee9146948319b8d37434b630 [term 1 LEADER]: Becoming Leader. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Running, Role: LEADER
I20240503 02:16:19.622934 19168 consensus_queue.cc:237] T 6bc5c6531d5144bf80f1b388f23703e9 P 836b43a7ee9146948319b8d37434b630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:19.624732 17623 catalog_manager.cc:5484] T 0f8c3cd667594af4969548ba820e0164 P 1cd8517d70d94ae494e713df7eb61b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3). New cstate: current_term: 1 leader_uuid: "1cd8517d70d94ae494e713df7eb61b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } }
I20240503 02:16:19.627452 17618 catalog_manager.cc:5484] T 6bc5c6531d5144bf80f1b388f23703e9 P 836b43a7ee9146948319b8d37434b630 reported cstate change: term changed from 0 to 1, leader changed from <none> to 836b43a7ee9146948319b8d37434b630 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "836b43a7ee9146948319b8d37434b630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } }
W20240503 02:16:20.005121 19205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:16:20.005519 19205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:16:20.012751 19205 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:16:20.033711 18880 consensus_queue.cc:1035] T 0f8c3cd667594af4969548ba820e0164 P 1cd8517d70d94ae494e713df7eb61b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:16:20.046576 18880 consensus_queue.cc:1035] T 0f8c3cd667594af4969548ba820e0164 P 1cd8517d70d94ae494e713df7eb61b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_5'
I20240503 02:16:20.062110 19168 consensus_queue.cc:1035] T 6bc5c6531d5144bf80f1b388f23703e9 P 836b43a7ee9146948319b8d37434b630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:16:20.095249 19168 consensus_queue.cc:1035] T 6bc5c6531d5144bf80f1b388f23703e9 P 836b43a7ee9146948319b8d37434b630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:16:20.135792 18880 consensus_queue.cc:1035] T 0122399369d14e57a55a1415c710e72f P 1cd8517d70d94ae494e713df7eb61b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:16:20.174865 17895 raft_consensus.cc:1261] T 0122399369d14e57a55a1415c710e72f P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Refusing update from remote peer 1cd8517d70d94ae494e713df7eb61b12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:16:20.175472 18880 consensus_queue.cc:1035] T 0122399369d14e57a55a1415c710e72f P 1cd8517d70d94ae494e713df7eb61b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 225.693 ms
time per row: 0.112846 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 02:16:21.321877 17622 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44922:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:16:21.330704 17744 tablet_service.cc:1467] Processing CreateTablet for tablet dacfa7037bae46fb9b3d34ba4b8e54b4 (DEFAULT_TABLE table=rebalance_test_table_6 [id=33fb920a8157448dbdca779e63857e83]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:21.331189 17744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dacfa7037bae46fb9b3d34ba4b8e54b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:21.331653 18003 tablet_service.cc:1467] Processing CreateTablet for tablet c5d3ec79ed6f48b085284b2c06fab3ec (DEFAULT_TABLE table=rebalance_test_table_6 [id=33fb920a8157448dbdca779e63857e83]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:21.332027 18003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5d3ec79ed6f48b085284b2c06fab3ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:21.332728 18135 tablet_service.cc:1467] Processing CreateTablet for tablet dacfa7037bae46fb9b3d34ba4b8e54b4 (DEFAULT_TABLE table=rebalance_test_table_6 [id=33fb920a8157448dbdca779e63857e83]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:21.332885 18133 tablet_service.cc:1467] Processing CreateTablet for tablet c5d3ec79ed6f48b085284b2c06fab3ec (DEFAULT_TABLE table=rebalance_test_table_6 [id=33fb920a8157448dbdca779e63857e83]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:21.333182 18135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dacfa7037bae46fb9b3d34ba4b8e54b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:21.333516 18134 tablet_service.cc:1467] Processing CreateTablet for tablet bcd9329d71504749b82738119bf7d3ef (DEFAULT_TABLE table=rebalance_test_table_6 [id=33fb920a8157448dbdca779e63857e83]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:21.333840 18134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcd9329d71504749b82738119bf7d3ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:21.336652 17874 tablet_service.cc:1467] Processing CreateTablet for tablet dacfa7037bae46fb9b3d34ba4b8e54b4 (DEFAULT_TABLE table=rebalance_test_table_6 [id=33fb920a8157448dbdca779e63857e83]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:21.337133 17874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dacfa7037bae46fb9b3d34ba4b8e54b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:21.337424 17873 tablet_service.cc:1467] Processing CreateTablet for tablet bcd9329d71504749b82738119bf7d3ef (DEFAULT_TABLE table=rebalance_test_table_6 [id=33fb920a8157448dbdca779e63857e83]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:21.337790 17873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcd9329d71504749b82738119bf7d3ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:21.338114 17743 tablet_service.cc:1467] Processing CreateTablet for tablet c5d3ec79ed6f48b085284b2c06fab3ec (DEFAULT_TABLE table=rebalance_test_table_6 [id=33fb920a8157448dbdca779e63857e83]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:21.338447 17743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5d3ec79ed6f48b085284b2c06fab3ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:21.341430 17742 tablet_service.cc:1467] Processing CreateTablet for tablet bcd9329d71504749b82738119bf7d3ef (DEFAULT_TABLE table=rebalance_test_table_6 [id=33fb920a8157448dbdca779e63857e83]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:21.341773 17742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcd9329d71504749b82738119bf7d3ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:21.345646 18133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5d3ec79ed6f48b085284b2c06fab3ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:21.349321 19272 tablet_bootstrap.cc:492] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 4325d7d0ed484ad78a6ee54e9a003431: Bootstrap starting.
I20240503 02:16:21.352133 19272 tablet_bootstrap.cc:654] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 4325d7d0ed484ad78a6ee54e9a003431: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:21.360024 19275 tablet_bootstrap.cc:492] T c5d3ec79ed6f48b085284b2c06fab3ec P 1cd8517d70d94ae494e713df7eb61b12: Bootstrap starting.
I20240503 02:16:21.361115 19273 tablet_bootstrap.cc:492] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 0f3838ee7ad04e94aa65ac750dad0833: Bootstrap starting.
I20240503 02:16:21.361199 19274 tablet_bootstrap.cc:492] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 836b43a7ee9146948319b8d37434b630: Bootstrap starting.
I20240503 02:16:21.362733 19274 tablet_bootstrap.cc:654] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 836b43a7ee9146948319b8d37434b630: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:21.363270 19275 tablet_bootstrap.cc:654] T c5d3ec79ed6f48b085284b2c06fab3ec P 1cd8517d70d94ae494e713df7eb61b12: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:21.363422 19273 tablet_bootstrap.cc:654] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 0f3838ee7ad04e94aa65ac750dad0833: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:21.364799 19272 tablet_bootstrap.cc:492] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 4325d7d0ed484ad78a6ee54e9a003431: No bootstrap required, opened a new log
I20240503 02:16:21.365126 19272 ts_tablet_manager.cc:1367] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 4325d7d0ed484ad78a6ee54e9a003431: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.003s
I20240503 02:16:21.366046 19273 tablet_bootstrap.cc:492] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 0f3838ee7ad04e94aa65ac750dad0833: No bootstrap required, opened a new log
I20240503 02:16:21.366111 19274 tablet_bootstrap.cc:492] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 836b43a7ee9146948319b8d37434b630: No bootstrap required, opened a new log
I20240503 02:16:21.365998 19272 raft_consensus.cc:348] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:21.366182 19273 ts_tablet_manager.cc:1367] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 0f3838ee7ad04e94aa65ac750dad0833: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240503 02:16:21.366197 19272 raft_consensus.cc:374] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:21.366263 19274 ts_tablet_manager.cc:1367] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 836b43a7ee9146948319b8d37434b630: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:16:21.366308 19272 raft_consensus.cc:729] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4325d7d0ed484ad78a6ee54e9a003431, State: Initialized, Role: FOLLOWER
I20240503 02:16:21.366567 19272 consensus_queue.cc:260] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 4325d7d0ed484ad78a6ee54e9a003431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:21.366910 19273 raft_consensus.cc:348] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:21.367102 19273 raft_consensus.cc:374] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:21.367130 19272 ts_tablet_manager.cc:1398] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 4325d7d0ed484ad78a6ee54e9a003431: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:16:21.367194 19273 raft_consensus.cc:729] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Initialized, Role: FOLLOWER
I20240503 02:16:21.367108 19274 raft_consensus.cc:348] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:21.367305 19274 raft_consensus.cc:374] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:21.367389 19274 raft_consensus.cc:729] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Initialized, Role: FOLLOWER
I20240503 02:16:21.367447 19272 tablet_bootstrap.cc:492] T bcd9329d71504749b82738119bf7d3ef P 4325d7d0ed484ad78a6ee54e9a003431: Bootstrap starting.
I20240503 02:16:21.367406 19273 consensus_queue.cc:260] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:21.367983 19273 ts_tablet_manager.cc:1398] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 0f3838ee7ad04e94aa65ac750dad0833: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:16:21.367599 19274 consensus_queue.cc:260] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 836b43a7ee9146948319b8d37434b630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:21.368256 19273 tablet_bootstrap.cc:492] T bcd9329d71504749b82738119bf7d3ef P 0f3838ee7ad04e94aa65ac750dad0833: Bootstrap starting.
I20240503 02:16:21.368642 19274 ts_tablet_manager.cc:1398] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 836b43a7ee9146948319b8d37434b630: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:16:21.368906 19274 tablet_bootstrap.cc:492] T bcd9329d71504749b82738119bf7d3ef P 836b43a7ee9146948319b8d37434b630: Bootstrap starting.
I20240503 02:16:21.369773 19273 tablet_bootstrap.cc:654] T bcd9329d71504749b82738119bf7d3ef P 0f3838ee7ad04e94aa65ac750dad0833: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:21.371699 19272 tablet_bootstrap.cc:654] T bcd9329d71504749b82738119bf7d3ef P 4325d7d0ed484ad78a6ee54e9a003431: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:21.372619 19274 tablet_bootstrap.cc:654] T bcd9329d71504749b82738119bf7d3ef P 836b43a7ee9146948319b8d37434b630: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:21.373354 19275 tablet_bootstrap.cc:492] T c5d3ec79ed6f48b085284b2c06fab3ec P 1cd8517d70d94ae494e713df7eb61b12: No bootstrap required, opened a new log
I20240503 02:16:21.374419 19275 ts_tablet_manager.cc:1367] T c5d3ec79ed6f48b085284b2c06fab3ec P 1cd8517d70d94ae494e713df7eb61b12: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240503 02:16:21.375586 19275 raft_consensus.cc:348] T c5d3ec79ed6f48b085284b2c06fab3ec P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:21.377035 19275 raft_consensus.cc:374] T c5d3ec79ed6f48b085284b2c06fab3ec P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:21.377362 19275 raft_consensus.cc:729] T c5d3ec79ed6f48b085284b2c06fab3ec P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Initialized, Role: FOLLOWER
I20240503 02:16:21.377964 19275 consensus_queue.cc:260] T c5d3ec79ed6f48b085284b2c06fab3ec P 1cd8517d70d94ae494e713df7eb61b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:21.378481 19274 tablet_bootstrap.cc:492] T bcd9329d71504749b82738119bf7d3ef P 836b43a7ee9146948319b8d37434b630: No bootstrap required, opened a new log
I20240503 02:16:21.378677 19274 ts_tablet_manager.cc:1367] T bcd9329d71504749b82738119bf7d3ef P 836b43a7ee9146948319b8d37434b630: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20240503 02:16:21.378706 19275 ts_tablet_manager.cc:1398] T c5d3ec79ed6f48b085284b2c06fab3ec P 1cd8517d70d94ae494e713df7eb61b12: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:16:21.380434 19274 raft_consensus.cc:348] T bcd9329d71504749b82738119bf7d3ef P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:21.380683 19274 raft_consensus.cc:374] T bcd9329d71504749b82738119bf7d3ef P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:21.380789 19274 raft_consensus.cc:729] T bcd9329d71504749b82738119bf7d3ef P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Initialized, Role: FOLLOWER
I20240503 02:16:21.380893 19272 tablet_bootstrap.cc:492] T bcd9329d71504749b82738119bf7d3ef P 4325d7d0ed484ad78a6ee54e9a003431: No bootstrap required, opened a new log
I20240503 02:16:21.381129 19272 ts_tablet_manager.cc:1367] T bcd9329d71504749b82738119bf7d3ef P 4325d7d0ed484ad78a6ee54e9a003431: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20240503 02:16:21.381104 19274 consensus_queue.cc:260] T bcd9329d71504749b82738119bf7d3ef P 836b43a7ee9146948319b8d37434b630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:21.381767 19274 ts_tablet_manager.cc:1398] T bcd9329d71504749b82738119bf7d3ef P 836b43a7ee9146948319b8d37434b630: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:16:21.382007 19272 raft_consensus.cc:348] T bcd9329d71504749b82738119bf7d3ef P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:21.382304 19272 raft_consensus.cc:374] T bcd9329d71504749b82738119bf7d3ef P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:21.382359 19274 tablet_bootstrap.cc:492] T c5d3ec79ed6f48b085284b2c06fab3ec P 836b43a7ee9146948319b8d37434b630: Bootstrap starting.
I20240503 02:16:21.382431 19272 raft_consensus.cc:729] T bcd9329d71504749b82738119bf7d3ef P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4325d7d0ed484ad78a6ee54e9a003431, State: Initialized, Role: FOLLOWER
I20240503 02:16:21.382745 19272 consensus_queue.cc:260] T bcd9329d71504749b82738119bf7d3ef P 4325d7d0ed484ad78a6ee54e9a003431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:21.382999 19273 tablet_bootstrap.cc:492] T bcd9329d71504749b82738119bf7d3ef P 0f3838ee7ad04e94aa65ac750dad0833: No bootstrap required, opened a new log
I20240503 02:16:21.383175 19273 ts_tablet_manager.cc:1367] T bcd9329d71504749b82738119bf7d3ef P 0f3838ee7ad04e94aa65ac750dad0833: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.001s
I20240503 02:16:21.383472 19272 ts_tablet_manager.cc:1398] T bcd9329d71504749b82738119bf7d3ef P 4325d7d0ed484ad78a6ee54e9a003431: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:16:21.383767 19272 tablet_bootstrap.cc:492] T c5d3ec79ed6f48b085284b2c06fab3ec P 4325d7d0ed484ad78a6ee54e9a003431: Bootstrap starting.
I20240503 02:16:21.383949 19274 tablet_bootstrap.cc:654] T c5d3ec79ed6f48b085284b2c06fab3ec P 836b43a7ee9146948319b8d37434b630: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:21.383949 19273 raft_consensus.cc:348] T bcd9329d71504749b82738119bf7d3ef P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:21.384299 19273 raft_consensus.cc:374] T bcd9329d71504749b82738119bf7d3ef P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:21.384414 19273 raft_consensus.cc:729] T bcd9329d71504749b82738119bf7d3ef P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Initialized, Role: FOLLOWER
I20240503 02:16:21.384711 19273 consensus_queue.cc:260] T bcd9329d71504749b82738119bf7d3ef P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:21.385434 19273 ts_tablet_manager.cc:1398] T bcd9329d71504749b82738119bf7d3ef P 0f3838ee7ad04e94aa65ac750dad0833: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:16:21.393043 19274 tablet_bootstrap.cc:492] T c5d3ec79ed6f48b085284b2c06fab3ec P 836b43a7ee9146948319b8d37434b630: No bootstrap required, opened a new log
I20240503 02:16:21.393265 19274 ts_tablet_manager.cc:1367] T c5d3ec79ed6f48b085284b2c06fab3ec P 836b43a7ee9146948319b8d37434b630: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20240503 02:16:21.393508 19272 tablet_bootstrap.cc:654] T c5d3ec79ed6f48b085284b2c06fab3ec P 4325d7d0ed484ad78a6ee54e9a003431: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:21.394137 19274 raft_consensus.cc:348] T c5d3ec79ed6f48b085284b2c06fab3ec P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:21.394425 19274 raft_consensus.cc:374] T c5d3ec79ed6f48b085284b2c06fab3ec P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:21.394582 19274 raft_consensus.cc:729] T c5d3ec79ed6f48b085284b2c06fab3ec P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Initialized, Role: FOLLOWER
I20240503 02:16:21.394904 19274 consensus_queue.cc:260] T c5d3ec79ed6f48b085284b2c06fab3ec P 836b43a7ee9146948319b8d37434b630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:21.396286 19274 ts_tablet_manager.cc:1398] T c5d3ec79ed6f48b085284b2c06fab3ec P 836b43a7ee9146948319b8d37434b630: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:16:21.398981 19272 tablet_bootstrap.cc:492] T c5d3ec79ed6f48b085284b2c06fab3ec P 4325d7d0ed484ad78a6ee54e9a003431: No bootstrap required, opened a new log
I20240503 02:16:21.399135 19272 ts_tablet_manager.cc:1367] T c5d3ec79ed6f48b085284b2c06fab3ec P 4325d7d0ed484ad78a6ee54e9a003431: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20240503 02:16:21.399785 19272 raft_consensus.cc:348] T c5d3ec79ed6f48b085284b2c06fab3ec P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:21.399984 19272 raft_consensus.cc:374] T c5d3ec79ed6f48b085284b2c06fab3ec P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:21.400072 19272 raft_consensus.cc:729] T c5d3ec79ed6f48b085284b2c06fab3ec P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4325d7d0ed484ad78a6ee54e9a003431, State: Initialized, Role: FOLLOWER
I20240503 02:16:21.400310 19272 consensus_queue.cc:260] T c5d3ec79ed6f48b085284b2c06fab3ec P 4325d7d0ed484ad78a6ee54e9a003431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:21.400827 19272 ts_tablet_manager.cc:1398] T c5d3ec79ed6f48b085284b2c06fab3ec P 4325d7d0ed484ad78a6ee54e9a003431: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:16:21.475101 19267 raft_consensus.cc:482] T c5d3ec79ed6f48b085284b2c06fab3ec P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:21.475294 19267 raft_consensus.cc:504] T c5d3ec79ed6f48b085284b2c06fab3ec P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:21.475937 19267 leader_election.cc:288] T c5d3ec79ed6f48b085284b2c06fab3ec P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837), 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703)
I20240503 02:16:21.476328 17764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d3ec79ed6f48b085284b2c06fab3ec" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836b43a7ee9146948319b8d37434b630" is_pre_election: true
I20240503 02:16:21.476403 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d3ec79ed6f48b085284b2c06fab3ec" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431" is_pre_election: true
I20240503 02:16:21.476536 17764 raft_consensus.cc:2454] T c5d3ec79ed6f48b085284b2c06fab3ec P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cd8517d70d94ae494e713df7eb61b12 in term 0.
I20240503 02:16:21.476660 18155 raft_consensus.cc:2454] T c5d3ec79ed6f48b085284b2c06fab3ec P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cd8517d70d94ae494e713df7eb61b12 in term 0.
I20240503 02:16:21.476967 17958 leader_election.cc:302] T c5d3ec79ed6f48b085284b2c06fab3ec P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cd8517d70d94ae494e713df7eb61b12, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:21.477260 19267 raft_consensus.cc:2781] T c5d3ec79ed6f48b085284b2c06fab3ec P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:16:21.477370 19267 raft_consensus.cc:482] T c5d3ec79ed6f48b085284b2c06fab3ec P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:16:21.477450 19267 raft_consensus.cc:3037] T c5d3ec79ed6f48b085284b2c06fab3ec P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:21.479291 19267 raft_consensus.cc:504] T c5d3ec79ed6f48b085284b2c06fab3ec P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:21.479760 19267 leader_election.cc:288] T c5d3ec79ed6f48b085284b2c06fab3ec P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 1 election: Requested vote from peers 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837), 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703)
I20240503 02:16:21.480190 17764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d3ec79ed6f48b085284b2c06fab3ec" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836b43a7ee9146948319b8d37434b630"
I20240503 02:16:21.480276 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d3ec79ed6f48b085284b2c06fab3ec" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431"
I20240503 02:16:21.480432 17764 raft_consensus.cc:3037] T c5d3ec79ed6f48b085284b2c06fab3ec P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:21.480486 18155 raft_consensus.cc:3037] T c5d3ec79ed6f48b085284b2c06fab3ec P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:21.483286 17764 raft_consensus.cc:2454] T c5d3ec79ed6f48b085284b2c06fab3ec P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cd8517d70d94ae494e713df7eb61b12 in term 1.
I20240503 02:16:21.483278 18155 raft_consensus.cc:2454] T c5d3ec79ed6f48b085284b2c06fab3ec P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cd8517d70d94ae494e713df7eb61b12 in term 1.
I20240503 02:16:21.483664 17958 leader_election.cc:302] T c5d3ec79ed6f48b085284b2c06fab3ec P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cd8517d70d94ae494e713df7eb61b12, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:21.483925 19267 raft_consensus.cc:2781] T c5d3ec79ed6f48b085284b2c06fab3ec P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:16:21.484282 19267 raft_consensus.cc:686] T c5d3ec79ed6f48b085284b2c06fab3ec P 1cd8517d70d94ae494e713df7eb61b12 [term 1 LEADER]: Becoming Leader. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Running, Role: LEADER
I20240503 02:16:21.484588 19267 consensus_queue.cc:237] T c5d3ec79ed6f48b085284b2c06fab3ec P 1cd8517d70d94ae494e713df7eb61b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:21.487480 17622 catalog_manager.cc:5484] T c5d3ec79ed6f48b085284b2c06fab3ec P 1cd8517d70d94ae494e713df7eb61b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3). New cstate: current_term: 1 leader_uuid: "1cd8517d70d94ae494e713df7eb61b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } }
I20240503 02:16:21.534476 19247 raft_consensus.cc:482] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:21.534642 19247 raft_consensus.cc:504] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:21.535187 19247 leader_election.cc:288] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 4325d7d0ed484ad78a6ee54e9a003431 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837), 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647)
I20240503 02:16:21.535617 17764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacfa7037bae46fb9b3d34ba4b8e54b4" candidate_uuid: "4325d7d0ed484ad78a6ee54e9a003431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836b43a7ee9146948319b8d37434b630" is_pre_election: true
I20240503 02:16:21.535660 17893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacfa7037bae46fb9b3d34ba4b8e54b4" candidate_uuid: "4325d7d0ed484ad78a6ee54e9a003431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833" is_pre_election: true
I20240503 02:16:21.535820 17764 raft_consensus.cc:2454] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4325d7d0ed484ad78a6ee54e9a003431 in term 0.
I20240503 02:16:21.535856 17893 raft_consensus.cc:2454] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4325d7d0ed484ad78a6ee54e9a003431 in term 0.
I20240503 02:16:21.536183 18088 leader_election.cc:302] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 4325d7d0ed484ad78a6ee54e9a003431 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4325d7d0ed484ad78a6ee54e9a003431, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:21.536465 19247 raft_consensus.cc:2781] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:16:21.536571 19247 raft_consensus.cc:482] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:16:21.536664 19247 raft_consensus.cc:3037] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:21.538640 19247 raft_consensus.cc:504] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:21.539245 19247 leader_election.cc:288] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 4325d7d0ed484ad78a6ee54e9a003431 [CANDIDATE]: Term 1 election: Requested vote from peers 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837), 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647)
I20240503 02:16:21.539620 17764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacfa7037bae46fb9b3d34ba4b8e54b4" candidate_uuid: "4325d7d0ed484ad78a6ee54e9a003431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836b43a7ee9146948319b8d37434b630"
I20240503 02:16:21.539705 17893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacfa7037bae46fb9b3d34ba4b8e54b4" candidate_uuid: "4325d7d0ed484ad78a6ee54e9a003431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
I20240503 02:16:21.539791 17764 raft_consensus.cc:3037] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:21.539901 17893 raft_consensus.cc:3037] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:21.541682 17764 raft_consensus.cc:2454] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4325d7d0ed484ad78a6ee54e9a003431 in term 1.
I20240503 02:16:21.541833 17893 raft_consensus.cc:2454] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4325d7d0ed484ad78a6ee54e9a003431 in term 1.
I20240503 02:16:21.542094 18088 leader_election.cc:302] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 4325d7d0ed484ad78a6ee54e9a003431 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4325d7d0ed484ad78a6ee54e9a003431, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:21.542326 19247 raft_consensus.cc:2781] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:16:21.542690 19247 raft_consensus.cc:686] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 LEADER]: Becoming Leader. State: Replica: 4325d7d0ed484ad78a6ee54e9a003431, State: Running, Role: LEADER
I20240503 02:16:21.542953 19247 consensus_queue.cc:237] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 4325d7d0ed484ad78a6ee54e9a003431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:21.545861 17622 catalog_manager.cc:5484] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 4325d7d0ed484ad78a6ee54e9a003431 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4). New cstate: current_term: 1 leader_uuid: "4325d7d0ed484ad78a6ee54e9a003431" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } }
I20240503 02:16:21.577754 19128 raft_consensus.cc:482] T bcd9329d71504749b82738119bf7d3ef P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:21.577929 19128 raft_consensus.cc:504] T bcd9329d71504749b82738119bf7d3ef P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:21.578469 19128 leader_election.cc:288] T bcd9329d71504749b82738119bf7d3ef P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703), 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837)
I20240503 02:16:21.578919 17764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd9329d71504749b82738119bf7d3ef" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836b43a7ee9146948319b8d37434b630" is_pre_election: true
I20240503 02:16:21.578924 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd9329d71504749b82738119bf7d3ef" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431" is_pre_election: true
I20240503 02:16:21.579139 17764 raft_consensus.cc:2454] T bcd9329d71504749b82738119bf7d3ef P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 0.
I20240503 02:16:21.579138 18155 raft_consensus.cc:2454] T bcd9329d71504749b82738119bf7d3ef P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 0.
I20240503 02:16:21.579483 17828 leader_election.cc:302] T bcd9329d71504749b82738119bf7d3ef P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 4325d7d0ed484ad78a6ee54e9a003431; no voters:
I20240503 02:16:21.579711 19128 raft_consensus.cc:2781] T bcd9329d71504749b82738119bf7d3ef P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:16:21.579834 19128 raft_consensus.cc:482] T bcd9329d71504749b82738119bf7d3ef P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:16:21.579934 19128 raft_consensus.cc:3037] T bcd9329d71504749b82738119bf7d3ef P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:21.581763 19128 raft_consensus.cc:504] T bcd9329d71504749b82738119bf7d3ef P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:21.582248 19128 leader_election.cc:288] T bcd9329d71504749b82738119bf7d3ef P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 1 election: Requested vote from peers 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703), 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837)
I20240503 02:16:21.582705 17764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd9329d71504749b82738119bf7d3ef" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836b43a7ee9146948319b8d37434b630"
I20240503 02:16:21.582726 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd9329d71504749b82738119bf7d3ef" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431"
I20240503 02:16:21.582938 18155 raft_consensus.cc:3037] T bcd9329d71504749b82738119bf7d3ef P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:21.582942 17764 raft_consensus.cc:3037] T bcd9329d71504749b82738119bf7d3ef P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:21.585757 17764 raft_consensus.cc:2454] T bcd9329d71504749b82738119bf7d3ef P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 1.
I20240503 02:16:21.585780 18155 raft_consensus.cc:2454] T bcd9329d71504749b82738119bf7d3ef P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 1.
I20240503 02:16:21.586140 17828 leader_election.cc:302] T bcd9329d71504749b82738119bf7d3ef P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 4325d7d0ed484ad78a6ee54e9a003431; no voters:
I20240503 02:16:21.586409 19128 raft_consensus.cc:2781] T bcd9329d71504749b82738119bf7d3ef P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:16:21.586730 19128 raft_consensus.cc:686] T bcd9329d71504749b82738119bf7d3ef P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 LEADER]: Becoming Leader. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:21.587002 19128 consensus_queue.cc:237] T bcd9329d71504749b82738119bf7d3ef P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:21.589594 17622 catalog_manager.cc:5484] T bcd9329d71504749b82738119bf7d3ef P 0f3838ee7ad04e94aa65ac750dad0833 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2). New cstate: current_term: 1 leader_uuid: "0f3838ee7ad04e94aa65ac750dad0833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } }
W20240503 02:16:21.990556 19291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:16:21.990938 19291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:16:21.998637 19291 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:16:22.020045 19285 consensus_queue.cc:1035] T c5d3ec79ed6f48b085284b2c06fab3ec P 1cd8517d70d94ae494e713df7eb61b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_6'
I20240503 02:16:22.036808 19285 consensus_queue.cc:1035] T c5d3ec79ed6f48b085284b2c06fab3ec P 1cd8517d70d94ae494e713df7eb61b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:16:22.040551 19247 consensus_queue.cc:1035] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 4325d7d0ed484ad78a6ee54e9a003431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:16:22.052799 19314 consensus_queue.cc:1035] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 4325d7d0ed484ad78a6ee54e9a003431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:16:22.108880 19128 consensus_queue.cc:1035] T bcd9329d71504749b82738119bf7d3ef P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:16:22.113694 19128 consensus_queue.cc:1035] T bcd9329d71504749b82738119bf7d3ef P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 254.937 ms
time per row: 0.127468 ms
I20240503 02:16:22.424146 17617 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44922:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:16:22.432806 18003 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd3dd4b53b5463a8bc8c166ff355ebe (DEFAULT_TABLE table=rebalance_test_table_7 [id=366c1cb090ce46969a2f4980adc30fe1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:22.433353 18003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fd3dd4b53b5463a8bc8c166ff355ebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:22.433811 17744 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd3dd4b53b5463a8bc8c166ff355ebe (DEFAULT_TABLE table=rebalance_test_table_7 [id=366c1cb090ce46969a2f4980adc30fe1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:22.434252 17744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fd3dd4b53b5463a8bc8c166ff355ebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:22.435575 18135 tablet_service.cc:1467] Processing CreateTablet for tablet ae8c277b101444af870f4f9fd73d915b (DEFAULT_TABLE table=rebalance_test_table_7 [id=366c1cb090ce46969a2f4980adc30fe1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:22.435815 18133 tablet_service.cc:1467] Processing CreateTablet for tablet da0b26fe93614c818d195a45029d5b53 (DEFAULT_TABLE table=rebalance_test_table_7 [id=366c1cb090ce46969a2f4980adc30fe1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:22.436031 18135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae8c277b101444af870f4f9fd73d915b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:22.436544 18133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da0b26fe93614c818d195a45029d5b53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:22.438458 18005 tablet_service.cc:1467] Processing CreateTablet for tablet da0b26fe93614c818d195a45029d5b53 (DEFAULT_TABLE table=rebalance_test_table_7 [id=366c1cb090ce46969a2f4980adc30fe1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:22.438875 18005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da0b26fe93614c818d195a45029d5b53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:22.441318 17873 tablet_service.cc:1467] Processing CreateTablet for tablet ae8c277b101444af870f4f9fd73d915b (DEFAULT_TABLE table=rebalance_test_table_7 [id=366c1cb090ce46969a2f4980adc30fe1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:22.441390 18004 tablet_service.cc:1467] Processing CreateTablet for tablet ae8c277b101444af870f4f9fd73d915b (DEFAULT_TABLE table=rebalance_test_table_7 [id=366c1cb090ce46969a2f4980adc30fe1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:22.441712 18004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae8c277b101444af870f4f9fd73d915b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:22.441720 17873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae8c277b101444af870f4f9fd73d915b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:22.445660 17742 tablet_service.cc:1467] Processing CreateTablet for tablet da0b26fe93614c818d195a45029d5b53 (DEFAULT_TABLE table=rebalance_test_table_7 [id=366c1cb090ce46969a2f4980adc30fe1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:22.446054 17742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da0b26fe93614c818d195a45029d5b53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:22.448411 19359 tablet_bootstrap.cc:492] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 1cd8517d70d94ae494e713df7eb61b12: Bootstrap starting.
I20240503 02:16:22.449012 17874 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd3dd4b53b5463a8bc8c166ff355ebe (DEFAULT_TABLE table=rebalance_test_table_7 [id=366c1cb090ce46969a2f4980adc30fe1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:22.449481 17874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fd3dd4b53b5463a8bc8c166ff355ebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:22.450786 19359 tablet_bootstrap.cc:654] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 1cd8517d70d94ae494e713df7eb61b12: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:22.467094 19358 tablet_bootstrap.cc:492] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 836b43a7ee9146948319b8d37434b630: Bootstrap starting.
I20240503 02:16:22.468539 19362 tablet_bootstrap.cc:492] T ae8c277b101444af870f4f9fd73d915b P 0f3838ee7ad04e94aa65ac750dad0833: Bootstrap starting.
I20240503 02:16:22.470640 19362 tablet_bootstrap.cc:654] T ae8c277b101444af870f4f9fd73d915b P 0f3838ee7ad04e94aa65ac750dad0833: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:22.471030 19358 tablet_bootstrap.cc:654] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 836b43a7ee9146948319b8d37434b630: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:22.472241 19360 tablet_bootstrap.cc:492] T da0b26fe93614c818d195a45029d5b53 P 4325d7d0ed484ad78a6ee54e9a003431: Bootstrap starting.
I20240503 02:16:22.473321 19359 tablet_bootstrap.cc:492] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 1cd8517d70d94ae494e713df7eb61b12: No bootstrap required, opened a new log
I20240503 02:16:22.473486 19362 tablet_bootstrap.cc:492] T ae8c277b101444af870f4f9fd73d915b P 0f3838ee7ad04e94aa65ac750dad0833: No bootstrap required, opened a new log
I20240503 02:16:22.473541 19359 ts_tablet_manager.cc:1367] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 1cd8517d70d94ae494e713df7eb61b12: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.007s
I20240503 02:16:22.473659 19362 ts_tablet_manager.cc:1367] T ae8c277b101444af870f4f9fd73d915b P 0f3838ee7ad04e94aa65ac750dad0833: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 02:16:22.473990 19358 tablet_bootstrap.cc:492] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 836b43a7ee9146948319b8d37434b630: No bootstrap required, opened a new log
I20240503 02:16:22.474181 19358 ts_tablet_manager.cc:1367] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 836b43a7ee9146948319b8d37434b630: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 02:16:22.474467 19359 raft_consensus.cc:348] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:22.474509 19362 raft_consensus.cc:348] T ae8c277b101444af870f4f9fd73d915b P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:22.474682 19362 raft_consensus.cc:374] T ae8c277b101444af870f4f9fd73d915b P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:22.474782 19362 raft_consensus.cc:729] T ae8c277b101444af870f4f9fd73d915b P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Initialized, Role: FOLLOWER
I20240503 02:16:22.474802 19359 raft_consensus.cc:374] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:22.474927 19359 raft_consensus.cc:729] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Initialized, Role: FOLLOWER
I20240503 02:16:22.474925 19358 raft_consensus.cc:348] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:22.475188 19358 raft_consensus.cc:374] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:22.475128 19362 consensus_queue.cc:260] T ae8c277b101444af870f4f9fd73d915b P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:22.475320 19358 raft_consensus.cc:729] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Initialized, Role: FOLLOWER
I20240503 02:16:22.475237 19359 consensus_queue.cc:260] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 1cd8517d70d94ae494e713df7eb61b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:22.475862 19362 ts_tablet_manager.cc:1398] T ae8c277b101444af870f4f9fd73d915b P 0f3838ee7ad04e94aa65ac750dad0833: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:16:22.475864 19359 ts_tablet_manager.cc:1398] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 1cd8517d70d94ae494e713df7eb61b12: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:16:22.475709 19358 consensus_queue.cc:260] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 836b43a7ee9146948319b8d37434b630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:22.476545 19362 tablet_bootstrap.cc:492] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 0f3838ee7ad04e94aa65ac750dad0833: Bootstrap starting.
I20240503 02:16:22.477449 19358 ts_tablet_manager.cc:1398] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 836b43a7ee9146948319b8d37434b630: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:16:22.477775 19358 tablet_bootstrap.cc:492] T da0b26fe93614c818d195a45029d5b53 P 836b43a7ee9146948319b8d37434b630: Bootstrap starting.
I20240503 02:16:22.479359 19362 tablet_bootstrap.cc:654] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 0f3838ee7ad04e94aa65ac750dad0833: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:22.479390 19358 tablet_bootstrap.cc:654] T da0b26fe93614c818d195a45029d5b53 P 836b43a7ee9146948319b8d37434b630: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:22.481149 19360 tablet_bootstrap.cc:654] T da0b26fe93614c818d195a45029d5b53 P 4325d7d0ed484ad78a6ee54e9a003431: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:22.489818 19359 tablet_bootstrap.cc:492] T ae8c277b101444af870f4f9fd73d915b P 1cd8517d70d94ae494e713df7eb61b12: Bootstrap starting.
I20240503 02:16:22.491916 19359 tablet_bootstrap.cc:654] T ae8c277b101444af870f4f9fd73d915b P 1cd8517d70d94ae494e713df7eb61b12: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:22.492193 19360 tablet_bootstrap.cc:492] T da0b26fe93614c818d195a45029d5b53 P 4325d7d0ed484ad78a6ee54e9a003431: No bootstrap required, opened a new log
I20240503 02:16:22.492406 19360 ts_tablet_manager.cc:1367] T da0b26fe93614c818d195a45029d5b53 P 4325d7d0ed484ad78a6ee54e9a003431: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.005s
I20240503 02:16:22.493788 19360 raft_consensus.cc:348] T da0b26fe93614c818d195a45029d5b53 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:22.494170 19358 tablet_bootstrap.cc:492] T da0b26fe93614c818d195a45029d5b53 P 836b43a7ee9146948319b8d37434b630: No bootstrap required, opened a new log
I20240503 02:16:22.494352 19358 ts_tablet_manager.cc:1367] T da0b26fe93614c818d195a45029d5b53 P 836b43a7ee9146948319b8d37434b630: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.013s
I20240503 02:16:22.494889 19362 tablet_bootstrap.cc:492] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 0f3838ee7ad04e94aa65ac750dad0833: No bootstrap required, opened a new log
I20240503 02:16:22.495048 19362 ts_tablet_manager.cc:1367] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 0f3838ee7ad04e94aa65ac750dad0833: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.000s
I20240503 02:16:22.495147 19359 tablet_bootstrap.cc:492] T ae8c277b101444af870f4f9fd73d915b P 1cd8517d70d94ae494e713df7eb61b12: No bootstrap required, opened a new log
I20240503 02:16:22.495285 19359 ts_tablet_manager.cc:1367] T ae8c277b101444af870f4f9fd73d915b P 1cd8517d70d94ae494e713df7eb61b12: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20240503 02:16:22.495821 19362 raft_consensus.cc:348] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:22.495949 19359 raft_consensus.cc:348] T ae8c277b101444af870f4f9fd73d915b P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:22.496039 19362 raft_consensus.cc:374] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:22.496131 19362 raft_consensus.cc:729] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Initialized, Role: FOLLOWER
I20240503 02:16:22.496179 19359 raft_consensus.cc:374] T ae8c277b101444af870f4f9fd73d915b P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:22.496518 19362 consensus_queue.cc:260] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:22.496637 19359 raft_consensus.cc:729] T ae8c277b101444af870f4f9fd73d915b P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Initialized, Role: FOLLOWER
I20240503 02:16:22.496897 19359 consensus_queue.cc:260] T ae8c277b101444af870f4f9fd73d915b P 1cd8517d70d94ae494e713df7eb61b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:22.496940 19358 raft_consensus.cc:348] T da0b26fe93614c818d195a45029d5b53 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:22.497251 19358 raft_consensus.cc:374] T da0b26fe93614c818d195a45029d5b53 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:22.497305 19362 ts_tablet_manager.cc:1398] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 0f3838ee7ad04e94aa65ac750dad0833: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:16:22.497364 19358 raft_consensus.cc:729] T da0b26fe93614c818d195a45029d5b53 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Initialized, Role: FOLLOWER
I20240503 02:16:22.497640 19358 consensus_queue.cc:260] T da0b26fe93614c818d195a45029d5b53 P 836b43a7ee9146948319b8d37434b630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:22.498329 19358 ts_tablet_manager.cc:1398] T da0b26fe93614c818d195a45029d5b53 P 836b43a7ee9146948319b8d37434b630: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:16:22.497557 19360 raft_consensus.cc:374] T da0b26fe93614c818d195a45029d5b53 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:22.499464 19360 raft_consensus.cc:729] T da0b26fe93614c818d195a45029d5b53 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4325d7d0ed484ad78a6ee54e9a003431, State: Initialized, Role: FOLLOWER
I20240503 02:16:22.499781 19360 consensus_queue.cc:260] T da0b26fe93614c818d195a45029d5b53 P 4325d7d0ed484ad78a6ee54e9a003431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:22.500820 19360 ts_tablet_manager.cc:1398] T da0b26fe93614c818d195a45029d5b53 P 4325d7d0ed484ad78a6ee54e9a003431: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240503 02:16:22.501220 19360 tablet_bootstrap.cc:492] T ae8c277b101444af870f4f9fd73d915b P 4325d7d0ed484ad78a6ee54e9a003431: Bootstrap starting.
I20240503 02:16:22.502568 19359 ts_tablet_manager.cc:1398] T ae8c277b101444af870f4f9fd73d915b P 1cd8517d70d94ae494e713df7eb61b12: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:16:22.503576 19360 tablet_bootstrap.cc:654] T ae8c277b101444af870f4f9fd73d915b P 4325d7d0ed484ad78a6ee54e9a003431: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:22.504463 19359 tablet_bootstrap.cc:492] T da0b26fe93614c818d195a45029d5b53 P 1cd8517d70d94ae494e713df7eb61b12: Bootstrap starting.
I20240503 02:16:22.506690 19359 tablet_bootstrap.cc:654] T da0b26fe93614c818d195a45029d5b53 P 1cd8517d70d94ae494e713df7eb61b12: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:22.510022 19360 tablet_bootstrap.cc:492] T ae8c277b101444af870f4f9fd73d915b P 4325d7d0ed484ad78a6ee54e9a003431: No bootstrap required, opened a new log
I20240503 02:16:22.510191 19360 ts_tablet_manager.cc:1367] T ae8c277b101444af870f4f9fd73d915b P 4325d7d0ed484ad78a6ee54e9a003431: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20240503 02:16:22.510983 19360 raft_consensus.cc:348] T ae8c277b101444af870f4f9fd73d915b P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:22.511292 19360 raft_consensus.cc:374] T ae8c277b101444af870f4f9fd73d915b P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:22.511435 19360 raft_consensus.cc:729] T ae8c277b101444af870f4f9fd73d915b P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4325d7d0ed484ad78a6ee54e9a003431, State: Initialized, Role: FOLLOWER
I20240503 02:16:22.511757 19360 consensus_queue.cc:260] T ae8c277b101444af870f4f9fd73d915b P 4325d7d0ed484ad78a6ee54e9a003431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:22.512379 19360 ts_tablet_manager.cc:1398] T ae8c277b101444af870f4f9fd73d915b P 4325d7d0ed484ad78a6ee54e9a003431: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:16:22.514189 19359 tablet_bootstrap.cc:492] T da0b26fe93614c818d195a45029d5b53 P 1cd8517d70d94ae494e713df7eb61b12: No bootstrap required, opened a new log
I20240503 02:16:22.514361 19359 ts_tablet_manager.cc:1367] T da0b26fe93614c818d195a45029d5b53 P 1cd8517d70d94ae494e713df7eb61b12: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.008s
I20240503 02:16:22.515064 19359 raft_consensus.cc:348] T da0b26fe93614c818d195a45029d5b53 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:22.515256 19359 raft_consensus.cc:374] T da0b26fe93614c818d195a45029d5b53 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:22.515378 19359 raft_consensus.cc:729] T da0b26fe93614c818d195a45029d5b53 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Initialized, Role: FOLLOWER
I20240503 02:16:22.515650 19359 consensus_queue.cc:260] T da0b26fe93614c818d195a45029d5b53 P 1cd8517d70d94ae494e713df7eb61b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:22.516304 19359 ts_tablet_manager.cc:1398] T da0b26fe93614c818d195a45029d5b53 P 1cd8517d70d94ae494e713df7eb61b12: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:16:22.630250 19247 raft_consensus.cc:482] T ae8c277b101444af870f4f9fd73d915b P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:22.630432 19247 raft_consensus.cc:504] T ae8c277b101444af870f4f9fd73d915b P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:22.631096 19247 leader_election.cc:288] T ae8c277b101444af870f4f9fd73d915b P 4325d7d0ed484ad78a6ee54e9a003431 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759), 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647)
I20240503 02:16:22.631496 17895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae8c277b101444af870f4f9fd73d915b" candidate_uuid: "4325d7d0ed484ad78a6ee54e9a003431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833" is_pre_election: true
I20240503 02:16:22.631493 18025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae8c277b101444af870f4f9fd73d915b" candidate_uuid: "4325d7d0ed484ad78a6ee54e9a003431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd8517d70d94ae494e713df7eb61b12" is_pre_election: true
I20240503 02:16:22.631695 18025 raft_consensus.cc:2454] T ae8c277b101444af870f4f9fd73d915b P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4325d7d0ed484ad78a6ee54e9a003431 in term 0.
I20240503 02:16:22.631703 17895 raft_consensus.cc:2454] T ae8c277b101444af870f4f9fd73d915b P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4325d7d0ed484ad78a6ee54e9a003431 in term 0.
I20240503 02:16:22.632125 18088 leader_election.cc:302] T ae8c277b101444af870f4f9fd73d915b P 4325d7d0ed484ad78a6ee54e9a003431 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 4325d7d0ed484ad78a6ee54e9a003431; no voters:
I20240503 02:16:22.632375 19247 raft_consensus.cc:2781] T ae8c277b101444af870f4f9fd73d915b P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:16:22.632493 19247 raft_consensus.cc:482] T ae8c277b101444af870f4f9fd73d915b P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:16:22.632591 19247 raft_consensus.cc:3037] T ae8c277b101444af870f4f9fd73d915b P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:22.634986 19247 raft_consensus.cc:504] T ae8c277b101444af870f4f9fd73d915b P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:22.635592 19247 leader_election.cc:288] T ae8c277b101444af870f4f9fd73d915b P 4325d7d0ed484ad78a6ee54e9a003431 [CANDIDATE]: Term 1 election: Requested vote from peers 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759), 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647)
I20240503 02:16:22.635964 18025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae8c277b101444af870f4f9fd73d915b" candidate_uuid: "4325d7d0ed484ad78a6ee54e9a003431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:22.636068 17895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae8c277b101444af870f4f9fd73d915b" candidate_uuid: "4325d7d0ed484ad78a6ee54e9a003431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
I20240503 02:16:22.636200 18025 raft_consensus.cc:3037] T ae8c277b101444af870f4f9fd73d915b P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:22.636318 17895 raft_consensus.cc:3037] T ae8c277b101444af870f4f9fd73d915b P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:22.639147 18025 raft_consensus.cc:2454] T ae8c277b101444af870f4f9fd73d915b P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4325d7d0ed484ad78a6ee54e9a003431 in term 1.
I20240503 02:16:22.639146 17895 raft_consensus.cc:2454] T ae8c277b101444af870f4f9fd73d915b P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4325d7d0ed484ad78a6ee54e9a003431 in term 1.
I20240503 02:16:22.639542 18088 leader_election.cc:302] T ae8c277b101444af870f4f9fd73d915b P 4325d7d0ed484ad78a6ee54e9a003431 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 4325d7d0ed484ad78a6ee54e9a003431; no voters:
I20240503 02:16:22.639827 19247 raft_consensus.cc:2781] T ae8c277b101444af870f4f9fd73d915b P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:16:22.639963 19247 raft_consensus.cc:686] T ae8c277b101444af870f4f9fd73d915b P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 LEADER]: Becoming Leader. State: Replica: 4325d7d0ed484ad78a6ee54e9a003431, State: Running, Role: LEADER
I20240503 02:16:22.640215 19247 consensus_queue.cc:237] T ae8c277b101444af870f4f9fd73d915b P 4325d7d0ed484ad78a6ee54e9a003431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:22.643164 17621 catalog_manager.cc:5484] T ae8c277b101444af870f4f9fd73d915b P 4325d7d0ed484ad78a6ee54e9a003431 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4). New cstate: current_term: 1 leader_uuid: "4325d7d0ed484ad78a6ee54e9a003431" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } }
I20240503 02:16:22.650750 19171 raft_consensus.cc:482] T da0b26fe93614c818d195a45029d5b53 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:22.650929 19171 raft_consensus.cc:504] T da0b26fe93614c818d195a45029d5b53 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:22.651557 19171 leader_election.cc:288] T da0b26fe93614c818d195a45029d5b53 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759), 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703)
I20240503 02:16:22.651947 18025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da0b26fe93614c818d195a45029d5b53" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd8517d70d94ae494e713df7eb61b12" is_pre_election: true
I20240503 02:16:22.652021 18154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da0b26fe93614c818d195a45029d5b53" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431" is_pre_election: true
I20240503 02:16:22.652156 18025 raft_consensus.cc:2454] T da0b26fe93614c818d195a45029d5b53 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 0.
I20240503 02:16:22.652227 18154 raft_consensus.cc:2454] T da0b26fe93614c818d195a45029d5b53 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 0.
I20240503 02:16:22.652545 19171 raft_consensus.cc:482] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:22.652606 17700 leader_election.cc:302] T da0b26fe93614c818d195a45029d5b53 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cd8517d70d94ae494e713df7eb61b12, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:22.652727 19171 raft_consensus.cc:504] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:22.653271 19372 raft_consensus.cc:2781] T da0b26fe93614c818d195a45029d5b53 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:16:22.653381 19372 raft_consensus.cc:482] T da0b26fe93614c818d195a45029d5b53 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:16:22.653455 19171 leader_election.cc:288] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759), 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647)
I20240503 02:16:22.653553 19372 raft_consensus.cc:3037] T da0b26fe93614c818d195a45029d5b53 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:22.653904 18025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd3dd4b53b5463a8bc8c166ff355ebe" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd8517d70d94ae494e713df7eb61b12" is_pre_election: true
I20240503 02:16:22.653967 17895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd3dd4b53b5463a8bc8c166ff355ebe" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833" is_pre_election: true
I20240503 02:16:22.654127 18025 raft_consensus.cc:2454] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 0.
I20240503 02:16:22.654179 17895 raft_consensus.cc:2454] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 0.
I20240503 02:16:22.654599 17697 leader_election.cc:302] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:22.655344 19171 raft_consensus.cc:2781] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:16:22.655423 19171 raft_consensus.cc:482] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:16:22.655557 19171 raft_consensus.cc:3037] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:22.656045 19372 raft_consensus.cc:504] T da0b26fe93614c818d195a45029d5b53 P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:22.656610 19372 leader_election.cc:288] T da0b26fe93614c818d195a45029d5b53 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 1 election: Requested vote from peers 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759), 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703)
I20240503 02:16:22.656975 18025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da0b26fe93614c818d195a45029d5b53" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:22.657187 18025 raft_consensus.cc:3037] T da0b26fe93614c818d195a45029d5b53 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:22.657135 18154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da0b26fe93614c818d195a45029d5b53" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431"
I20240503 02:16:22.657328 18154 raft_consensus.cc:3037] T da0b26fe93614c818d195a45029d5b53 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:22.658057 19171 raft_consensus.cc:504] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:22.658921 18024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd3dd4b53b5463a8bc8c166ff355ebe" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:22.658985 18025 raft_consensus.cc:2454] T da0b26fe93614c818d195a45029d5b53 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 1.
I20240503 02:16:22.659107 18024 raft_consensus.cc:3037] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:22.660162 18154 raft_consensus.cc:2454] T da0b26fe93614c818d195a45029d5b53 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 1.
I20240503 02:16:22.660533 17697 leader_election.cc:302] T da0b26fe93614c818d195a45029d5b53 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4325d7d0ed484ad78a6ee54e9a003431, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:22.660781 19372 raft_consensus.cc:2781] T da0b26fe93614c818d195a45029d5b53 P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:16:22.661240 19372 raft_consensus.cc:686] T da0b26fe93614c818d195a45029d5b53 P 836b43a7ee9146948319b8d37434b630 [term 1 LEADER]: Becoming Leader. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Running, Role: LEADER
I20240503 02:16:22.661671 18024 raft_consensus.cc:2454] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 1.
I20240503 02:16:22.661568 19372 consensus_queue.cc:237] T da0b26fe93614c818d195a45029d5b53 P 836b43a7ee9146948319b8d37434b630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:22.662169 17700 leader_election.cc:302] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cd8517d70d94ae494e713df7eb61b12, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:22.662407 19373 raft_consensus.cc:2781] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:16:22.663098 17895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd3dd4b53b5463a8bc8c166ff355ebe" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
I20240503 02:16:22.663285 17895 raft_consensus.cc:3037] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:22.665356 17895 raft_consensus.cc:2454] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 1.
I20240503 02:16:22.665567 17621 catalog_manager.cc:5484] T da0b26fe93614c818d195a45029d5b53 P 836b43a7ee9146948319b8d37434b630 reported cstate change: term changed from 0 to 1, leader changed from <none> to 836b43a7ee9146948319b8d37434b630 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "836b43a7ee9146948319b8d37434b630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } }
I20240503 02:16:22.666164 19171 leader_election.cc:288] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 1 election: Requested vote from peers 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759), 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647)
I20240503 02:16:22.666719 19373 raft_consensus.cc:686] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 836b43a7ee9146948319b8d37434b630 [term 1 LEADER]: Becoming Leader. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Running, Role: LEADER
I20240503 02:16:22.666988 19373 consensus_queue.cc:237] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 836b43a7ee9146948319b8d37434b630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:22.671240 17621 catalog_manager.cc:5484] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 836b43a7ee9146948319b8d37434b630 reported cstate change: term changed from 0 to 1, leader changed from <none> to 836b43a7ee9146948319b8d37434b630 (127.22.108.1). New cstate: current_term: 1 leader_uuid: "836b43a7ee9146948319b8d37434b630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } }
I20240503 02:16:23.062214 19247 consensus_queue.cc:1035] T ae8c277b101444af870f4f9fd73d915b P 4325d7d0ed484ad78a6ee54e9a003431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:16:23.062641 19377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:16:23.063062 19377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 02:16:23.067976 19247 consensus_queue.cc:1035] T ae8c277b101444af870f4f9fd73d915b P 4325d7d0ed484ad78a6ee54e9a003431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:16:23.075677 19377 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240503 02:16:23.116902 19373 consensus_queue.cc:1035] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 836b43a7ee9146948319b8d37434b630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:16:23.123184 19373 consensus_queue.cc:1035] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 836b43a7ee9146948319b8d37434b630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:16:23.202966 19375 consensus_queue.cc:1035] T da0b26fe93614c818d195a45029d5b53 P 836b43a7ee9146948319b8d37434b630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:16:23.214746 19171 consensus_queue.cc:1035] T da0b26fe93614c818d195a45029d5b53 P 836b43a7ee9146948319b8d37434b630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 204.067 ms
time per row: 0.102034 ms
I20240503 02:16:23.463013 17617 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44922:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:16:23.472324 17744 tablet_service.cc:1467] Processing CreateTablet for tablet bb6aa5daccb6436fb7910d2d28244b7c (DEFAULT_TABLE table=rebalance_test_table_8 [id=c74eab6085af49b486512fbe753210fe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:23.472801 17744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb6aa5daccb6436fb7910d2d28244b7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:23.472967 18005 tablet_service.cc:1467] Processing CreateTablet for tablet bb6aa5daccb6436fb7910d2d28244b7c (DEFAULT_TABLE table=rebalance_test_table_8 [id=c74eab6085af49b486512fbe753210fe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:23.473456 18005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb6aa5daccb6436fb7910d2d28244b7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:23.473914 17873 tablet_service.cc:1467] Processing CreateTablet for tablet bb6aa5daccb6436fb7910d2d28244b7c (DEFAULT_TABLE table=rebalance_test_table_8 [id=c74eab6085af49b486512fbe753210fe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:23.474334 17873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb6aa5daccb6436fb7910d2d28244b7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:23.477383 19438 tablet_bootstrap.cc:492] T bb6aa5daccb6436fb7910d2d28244b7c P 836b43a7ee9146948319b8d37434b630: Bootstrap starting.
I20240503 02:16:23.477877 18004 tablet_service.cc:1467] Processing CreateTablet for tablet 954f9a466d8648958df7baa916ea8558 (DEFAULT_TABLE table=rebalance_test_table_8 [id=c74eab6085af49b486512fbe753210fe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:23.478229 18004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 954f9a466d8648958df7baa916ea8558. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:23.479164 19438 tablet_bootstrap.cc:654] T bb6aa5daccb6436fb7910d2d28244b7c P 836b43a7ee9146948319b8d37434b630: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:23.479293 18133 tablet_service.cc:1467] Processing CreateTablet for tablet 353ecf71ac51445087fa3f07e2836db5 (DEFAULT_TABLE table=rebalance_test_table_8 [id=c74eab6085af49b486512fbe753210fe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:23.479722 18133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 353ecf71ac51445087fa3f07e2836db5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:23.485514 18135 tablet_service.cc:1467] Processing CreateTablet for tablet 954f9a466d8648958df7baa916ea8558 (DEFAULT_TABLE table=rebalance_test_table_8 [id=c74eab6085af49b486512fbe753210fe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:23.485589 17874 tablet_service.cc:1467] Processing CreateTablet for tablet 353ecf71ac51445087fa3f07e2836db5 (DEFAULT_TABLE table=rebalance_test_table_8 [id=c74eab6085af49b486512fbe753210fe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:23.485956 17874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 353ecf71ac51445087fa3f07e2836db5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:23.485968 18135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 954f9a466d8648958df7baa916ea8558. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:23.489209 17742 tablet_service.cc:1467] Processing CreateTablet for tablet 353ecf71ac51445087fa3f07e2836db5 (DEFAULT_TABLE table=rebalance_test_table_8 [id=c74eab6085af49b486512fbe753210fe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:23.489401 19438 tablet_bootstrap.cc:492] T bb6aa5daccb6436fb7910d2d28244b7c P 836b43a7ee9146948319b8d37434b630: No bootstrap required, opened a new log
I20240503 02:16:23.489377 17743 tablet_service.cc:1467] Processing CreateTablet for tablet 954f9a466d8648958df7baa916ea8558 (DEFAULT_TABLE table=rebalance_test_table_8 [id=c74eab6085af49b486512fbe753210fe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:23.489571 19438 ts_tablet_manager.cc:1367] T bb6aa5daccb6436fb7910d2d28244b7c P 836b43a7ee9146948319b8d37434b630: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.007s
I20240503 02:16:23.489660 17742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 353ecf71ac51445087fa3f07e2836db5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:23.489884 17743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 954f9a466d8648958df7baa916ea8558. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:23.490507 19438 raft_consensus.cc:348] T bb6aa5daccb6436fb7910d2d28244b7c P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:23.490729 19438 raft_consensus.cc:374] T bb6aa5daccb6436fb7910d2d28244b7c P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:23.490859 19438 raft_consensus.cc:729] T bb6aa5daccb6436fb7910d2d28244b7c P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Initialized, Role: FOLLOWER
I20240503 02:16:23.491159 19438 consensus_queue.cc:260] T bb6aa5daccb6436fb7910d2d28244b7c P 836b43a7ee9146948319b8d37434b630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:23.493481 19438 ts_tablet_manager.cc:1398] T bb6aa5daccb6436fb7910d2d28244b7c P 836b43a7ee9146948319b8d37434b630: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:16:23.500464 19438 tablet_bootstrap.cc:492] T 353ecf71ac51445087fa3f07e2836db5 P 836b43a7ee9146948319b8d37434b630: Bootstrap starting.
I20240503 02:16:23.502301 19441 tablet_bootstrap.cc:492] T 954f9a466d8648958df7baa916ea8558 P 1cd8517d70d94ae494e713df7eb61b12: Bootstrap starting.
I20240503 02:16:23.503458 19438 tablet_bootstrap.cc:654] T 353ecf71ac51445087fa3f07e2836db5 P 836b43a7ee9146948319b8d37434b630: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:23.504117 19441 tablet_bootstrap.cc:654] T 954f9a466d8648958df7baa916ea8558 P 1cd8517d70d94ae494e713df7eb61b12: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:23.506537 19440 tablet_bootstrap.cc:492] T bb6aa5daccb6436fb7910d2d28244b7c P 0f3838ee7ad04e94aa65ac750dad0833: Bootstrap starting.
I20240503 02:16:23.506695 19442 tablet_bootstrap.cc:492] T 353ecf71ac51445087fa3f07e2836db5 P 4325d7d0ed484ad78a6ee54e9a003431: Bootstrap starting.
I20240503 02:16:23.508322 19442 tablet_bootstrap.cc:654] T 353ecf71ac51445087fa3f07e2836db5 P 4325d7d0ed484ad78a6ee54e9a003431: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:23.510787 19441 tablet_bootstrap.cc:492] T 954f9a466d8648958df7baa916ea8558 P 1cd8517d70d94ae494e713df7eb61b12: No bootstrap required, opened a new log
I20240503 02:16:23.511018 19441 ts_tablet_manager.cc:1367] T 954f9a466d8648958df7baa916ea8558 P 1cd8517d70d94ae494e713df7eb61b12: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20240503 02:16:23.512105 19441 raft_consensus.cc:348] T 954f9a466d8648958df7baa916ea8558 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:23.512408 19441 raft_consensus.cc:374] T 954f9a466d8648958df7baa916ea8558 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:23.512544 19441 raft_consensus.cc:729] T 954f9a466d8648958df7baa916ea8558 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Initialized, Role: FOLLOWER
I20240503 02:16:23.512838 19441 consensus_queue.cc:260] T 954f9a466d8648958df7baa916ea8558 P 1cd8517d70d94ae494e713df7eb61b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:23.514225 19441 ts_tablet_manager.cc:1398] T 954f9a466d8648958df7baa916ea8558 P 1cd8517d70d94ae494e713df7eb61b12: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:16:23.514537 19441 tablet_bootstrap.cc:492] T bb6aa5daccb6436fb7910d2d28244b7c P 1cd8517d70d94ae494e713df7eb61b12: Bootstrap starting.
I20240503 02:16:23.515806 19438 tablet_bootstrap.cc:492] T 353ecf71ac51445087fa3f07e2836db5 P 836b43a7ee9146948319b8d37434b630: No bootstrap required, opened a new log
I20240503 02:16:23.516036 19438 ts_tablet_manager.cc:1367] T 353ecf71ac51445087fa3f07e2836db5 P 836b43a7ee9146948319b8d37434b630: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.006s
I20240503 02:16:23.516177 19441 tablet_bootstrap.cc:654] T bb6aa5daccb6436fb7910d2d28244b7c P 1cd8517d70d94ae494e713df7eb61b12: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:23.516970 19438 raft_consensus.cc:348] T 353ecf71ac51445087fa3f07e2836db5 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:23.517378 19438 raft_consensus.cc:374] T 353ecf71ac51445087fa3f07e2836db5 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:23.517529 19438 raft_consensus.cc:729] T 353ecf71ac51445087fa3f07e2836db5 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Initialized, Role: FOLLOWER
I20240503 02:16:23.517957 19438 consensus_queue.cc:260] T 353ecf71ac51445087fa3f07e2836db5 P 836b43a7ee9146948319b8d37434b630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:23.518640 19438 ts_tablet_manager.cc:1398] T 353ecf71ac51445087fa3f07e2836db5 P 836b43a7ee9146948319b8d37434b630: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:16:23.518936 19440 tablet_bootstrap.cc:654] T bb6aa5daccb6436fb7910d2d28244b7c P 0f3838ee7ad04e94aa65ac750dad0833: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:23.519006 19438 tablet_bootstrap.cc:492] T 954f9a466d8648958df7baa916ea8558 P 836b43a7ee9146948319b8d37434b630: Bootstrap starting.
I20240503 02:16:23.523157 19438 tablet_bootstrap.cc:654] T 954f9a466d8648958df7baa916ea8558 P 836b43a7ee9146948319b8d37434b630: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:23.524679 19267 raft_consensus.cc:482] T 954f9a466d8648958df7baa916ea8558 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:23.524847 19267 raft_consensus.cc:504] T 954f9a466d8648958df7baa916ea8558 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:23.525494 19267 leader_election.cc:288] T 954f9a466d8648958df7baa916ea8558 P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703), 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837)
I20240503 02:16:23.526575 18154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "954f9a466d8648958df7baa916ea8558" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431" is_pre_election: true
I20240503 02:16:23.526575 17763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "954f9a466d8648958df7baa916ea8558" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836b43a7ee9146948319b8d37434b630" is_pre_election: true
W20240503 02:16:23.527642 17958 leader_election.cc:341] T 954f9a466d8648958df7baa916ea8558 P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:16:23.528004 17958 leader_election.cc:341] T 954f9a466d8648958df7baa916ea8558 P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:16:23.528139 17958 leader_election.cc:302] T 954f9a466d8648958df7baa916ea8558 P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cd8517d70d94ae494e713df7eb61b12; no voters: 4325d7d0ed484ad78a6ee54e9a003431, 836b43a7ee9146948319b8d37434b630
I20240503 02:16:23.528399 19442 tablet_bootstrap.cc:492] T 353ecf71ac51445087fa3f07e2836db5 P 4325d7d0ed484ad78a6ee54e9a003431: No bootstrap required, opened a new log
I20240503 02:16:23.528525 19442 ts_tablet_manager.cc:1367] T 353ecf71ac51445087fa3f07e2836db5 P 4325d7d0ed484ad78a6ee54e9a003431: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.000s
I20240503 02:16:23.529143 19267 raft_consensus.cc:2726] T 954f9a466d8648958df7baa916ea8558 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:16:23.529340 19442 raft_consensus.cc:348] T 353ecf71ac51445087fa3f07e2836db5 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:23.529727 19442 raft_consensus.cc:374] T 353ecf71ac51445087fa3f07e2836db5 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:23.529855 19442 raft_consensus.cc:729] T 353ecf71ac51445087fa3f07e2836db5 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4325d7d0ed484ad78a6ee54e9a003431, State: Initialized, Role: FOLLOWER
I20240503 02:16:23.530099 19442 consensus_queue.cc:260] T 353ecf71ac51445087fa3f07e2836db5 P 4325d7d0ed484ad78a6ee54e9a003431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:23.530829 19442 ts_tablet_manager.cc:1398] T 353ecf71ac51445087fa3f07e2836db5 P 4325d7d0ed484ad78a6ee54e9a003431: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:16:23.531172 19442 tablet_bootstrap.cc:492] T 954f9a466d8648958df7baa916ea8558 P 4325d7d0ed484ad78a6ee54e9a003431: Bootstrap starting.
I20240503 02:16:23.531168 19441 tablet_bootstrap.cc:492] T bb6aa5daccb6436fb7910d2d28244b7c P 1cd8517d70d94ae494e713df7eb61b12: No bootstrap required, opened a new log
I20240503 02:16:23.531359 19441 ts_tablet_manager.cc:1367] T bb6aa5daccb6436fb7910d2d28244b7c P 1cd8517d70d94ae494e713df7eb61b12: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.007s
I20240503 02:16:23.532377 19441 raft_consensus.cc:348] T bb6aa5daccb6436fb7910d2d28244b7c P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:23.532732 19441 raft_consensus.cc:374] T bb6aa5daccb6436fb7910d2d28244b7c P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:23.532876 19441 raft_consensus.cc:729] T bb6aa5daccb6436fb7910d2d28244b7c P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Initialized, Role: FOLLOWER
I20240503 02:16:23.532966 19442 tablet_bootstrap.cc:654] T 954f9a466d8648958df7baa916ea8558 P 4325d7d0ed484ad78a6ee54e9a003431: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:23.533278 19441 consensus_queue.cc:260] T bb6aa5daccb6436fb7910d2d28244b7c P 1cd8517d70d94ae494e713df7eb61b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:23.534216 19441 ts_tablet_manager.cc:1398] T bb6aa5daccb6436fb7910d2d28244b7c P 1cd8517d70d94ae494e713df7eb61b12: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:16:23.536820 19438 tablet_bootstrap.cc:492] T 954f9a466d8648958df7baa916ea8558 P 836b43a7ee9146948319b8d37434b630: No bootstrap required, opened a new log
I20240503 02:16:23.536967 19438 ts_tablet_manager.cc:1367] T 954f9a466d8648958df7baa916ea8558 P 836b43a7ee9146948319b8d37434b630: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.012s
I20240503 02:16:23.537786 19438 raft_consensus.cc:348] T 954f9a466d8648958df7baa916ea8558 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:23.537971 19438 raft_consensus.cc:374] T 954f9a466d8648958df7baa916ea8558 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:23.538077 19438 raft_consensus.cc:729] T 954f9a466d8648958df7baa916ea8558 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Initialized, Role: FOLLOWER
I20240503 02:16:23.538445 19438 consensus_queue.cc:260] T 954f9a466d8648958df7baa916ea8558 P 836b43a7ee9146948319b8d37434b630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:23.539142 19438 ts_tablet_manager.cc:1398] T 954f9a466d8648958df7baa916ea8558 P 836b43a7ee9146948319b8d37434b630: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:16:23.539384 19440 tablet_bootstrap.cc:492] T bb6aa5daccb6436fb7910d2d28244b7c P 0f3838ee7ad04e94aa65ac750dad0833: No bootstrap required, opened a new log
I20240503 02:16:23.539573 19440 ts_tablet_manager.cc:1367] T bb6aa5daccb6436fb7910d2d28244b7c P 0f3838ee7ad04e94aa65ac750dad0833: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.008s
I20240503 02:16:23.540411 19440 raft_consensus.cc:348] T bb6aa5daccb6436fb7910d2d28244b7c P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:23.540676 19440 raft_consensus.cc:374] T bb6aa5daccb6436fb7910d2d28244b7c P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:23.540786 19440 raft_consensus.cc:729] T bb6aa5daccb6436fb7910d2d28244b7c P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Initialized, Role: FOLLOWER
I20240503 02:16:23.541111 19440 consensus_queue.cc:260] T bb6aa5daccb6436fb7910d2d28244b7c P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:23.542943 19440 ts_tablet_manager.cc:1398] T bb6aa5daccb6436fb7910d2d28244b7c P 0f3838ee7ad04e94aa65ac750dad0833: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:16:23.543295 19440 tablet_bootstrap.cc:492] T 353ecf71ac51445087fa3f07e2836db5 P 0f3838ee7ad04e94aa65ac750dad0833: Bootstrap starting.
I20240503 02:16:23.544871 19442 tablet_bootstrap.cc:492] T 954f9a466d8648958df7baa916ea8558 P 4325d7d0ed484ad78a6ee54e9a003431: No bootstrap required, opened a new log
I20240503 02:16:23.545116 19442 ts_tablet_manager.cc:1367] T 954f9a466d8648958df7baa916ea8558 P 4325d7d0ed484ad78a6ee54e9a003431: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.004s
I20240503 02:16:23.545557 19440 tablet_bootstrap.cc:654] T 353ecf71ac51445087fa3f07e2836db5 P 0f3838ee7ad04e94aa65ac750dad0833: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:23.545971 19442 raft_consensus.cc:348] T 954f9a466d8648958df7baa916ea8558 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:23.546257 19442 raft_consensus.cc:374] T 954f9a466d8648958df7baa916ea8558 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:23.546373 19442 raft_consensus.cc:729] T 954f9a466d8648958df7baa916ea8558 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4325d7d0ed484ad78a6ee54e9a003431, State: Initialized, Role: FOLLOWER
I20240503 02:16:23.546756 19442 consensus_queue.cc:260] T 954f9a466d8648958df7baa916ea8558 P 4325d7d0ed484ad78a6ee54e9a003431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:23.547416 19442 ts_tablet_manager.cc:1398] T 954f9a466d8648958df7baa916ea8558 P 4325d7d0ed484ad78a6ee54e9a003431: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:16:23.548539 19440 tablet_bootstrap.cc:492] T 353ecf71ac51445087fa3f07e2836db5 P 0f3838ee7ad04e94aa65ac750dad0833: No bootstrap required, opened a new log
I20240503 02:16:23.548677 19440 ts_tablet_manager.cc:1367] T 353ecf71ac51445087fa3f07e2836db5 P 0f3838ee7ad04e94aa65ac750dad0833: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:16:23.549319 19440 raft_consensus.cc:348] T 353ecf71ac51445087fa3f07e2836db5 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:23.549553 19440 raft_consensus.cc:374] T 353ecf71ac51445087fa3f07e2836db5 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:23.549633 19440 raft_consensus.cc:729] T 353ecf71ac51445087fa3f07e2836db5 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Initialized, Role: FOLLOWER
I20240503 02:16:23.549882 19440 consensus_queue.cc:260] T 353ecf71ac51445087fa3f07e2836db5 P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:23.550433 19440 ts_tablet_manager.cc:1398] T 353ecf71ac51445087fa3f07e2836db5 P 0f3838ee7ad04e94aa65ac750dad0833: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:16:23.557135 19128 raft_consensus.cc:482] T bb6aa5daccb6436fb7910d2d28244b7c P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:23.557294 19128 raft_consensus.cc:504] T bb6aa5daccb6436fb7910d2d28244b7c P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:23.557806 19128 leader_election.cc:288] T bb6aa5daccb6436fb7910d2d28244b7c P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837), 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759)
I20240503 02:16:23.558264 17763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb6aa5daccb6436fb7910d2d28244b7c" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836b43a7ee9146948319b8d37434b630" is_pre_election: true
I20240503 02:16:23.558331 18025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb6aa5daccb6436fb7910d2d28244b7c" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd8517d70d94ae494e713df7eb61b12" is_pre_election: true
I20240503 02:16:23.558482 17763 raft_consensus.cc:2454] T bb6aa5daccb6436fb7910d2d28244b7c P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 0.
I20240503 02:16:23.558542 18025 raft_consensus.cc:2454] T bb6aa5daccb6436fb7910d2d28244b7c P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 0.
I20240503 02:16:23.558830 17828 leader_election.cc:302] T bb6aa5daccb6436fb7910d2d28244b7c P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:23.559120 19128 raft_consensus.cc:2781] T bb6aa5daccb6436fb7910d2d28244b7c P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:16:23.559201 19128 raft_consensus.cc:482] T bb6aa5daccb6436fb7910d2d28244b7c P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:16:23.559315 19128 raft_consensus.cc:3037] T bb6aa5daccb6436fb7910d2d28244b7c P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:23.561147 19128 raft_consensus.cc:504] T bb6aa5daccb6436fb7910d2d28244b7c P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:23.561650 19128 leader_election.cc:288] T bb6aa5daccb6436fb7910d2d28244b7c P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 1 election: Requested vote from peers 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837), 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759)
I20240503 02:16:23.561945 17763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb6aa5daccb6436fb7910d2d28244b7c" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836b43a7ee9146948319b8d37434b630"
I20240503 02:16:23.562023 18025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb6aa5daccb6436fb7910d2d28244b7c" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:23.562139 17763 raft_consensus.cc:3037] T bb6aa5daccb6436fb7910d2d28244b7c P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:23.562203 18025 raft_consensus.cc:3037] T bb6aa5daccb6436fb7910d2d28244b7c P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:23.564024 17763 raft_consensus.cc:2454] T bb6aa5daccb6436fb7910d2d28244b7c P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 1.
I20240503 02:16:23.564035 18025 raft_consensus.cc:2454] T bb6aa5daccb6436fb7910d2d28244b7c P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 1.
I20240503 02:16:23.564441 17831 leader_election.cc:302] T bb6aa5daccb6436fb7910d2d28244b7c P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 1cd8517d70d94ae494e713df7eb61b12; no voters:
I20240503 02:16:23.564671 19128 raft_consensus.cc:2781] T bb6aa5daccb6436fb7910d2d28244b7c P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:16:23.565114 19128 raft_consensus.cc:686] T bb6aa5daccb6436fb7910d2d28244b7c P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 LEADER]: Becoming Leader. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:23.565392 19128 consensus_queue.cc:237] T bb6aa5daccb6436fb7910d2d28244b7c P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:23.567966 17620 catalog_manager.cc:5484] T bb6aa5daccb6436fb7910d2d28244b7c P 0f3838ee7ad04e94aa65ac750dad0833 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2). New cstate: current_term: 1 leader_uuid: "0f3838ee7ad04e94aa65ac750dad0833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } }
I20240503 02:16:23.603622 19314 raft_consensus.cc:482] T 353ecf71ac51445087fa3f07e2836db5 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:23.603789 19314 raft_consensus.cc:504] T 353ecf71ac51445087fa3f07e2836db5 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:23.604348 19314 leader_election.cc:288] T 353ecf71ac51445087fa3f07e2836db5 P 4325d7d0ed484ad78a6ee54e9a003431 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837), 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647)
I20240503 02:16:23.604760 17763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "353ecf71ac51445087fa3f07e2836db5" candidate_uuid: "4325d7d0ed484ad78a6ee54e9a003431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836b43a7ee9146948319b8d37434b630" is_pre_election: true
I20240503 02:16:23.604836 17895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "353ecf71ac51445087fa3f07e2836db5" candidate_uuid: "4325d7d0ed484ad78a6ee54e9a003431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833" is_pre_election: true
I20240503 02:16:23.605010 17763 raft_consensus.cc:2454] T 353ecf71ac51445087fa3f07e2836db5 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4325d7d0ed484ad78a6ee54e9a003431 in term 0.
I20240503 02:16:23.605129 17895 raft_consensus.cc:2454] T 353ecf71ac51445087fa3f07e2836db5 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4325d7d0ed484ad78a6ee54e9a003431 in term 0.
I20240503 02:16:23.605446 18088 leader_election.cc:302] T 353ecf71ac51445087fa3f07e2836db5 P 4325d7d0ed484ad78a6ee54e9a003431 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4325d7d0ed484ad78a6ee54e9a003431, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:23.605698 19314 raft_consensus.cc:2781] T 353ecf71ac51445087fa3f07e2836db5 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:16:23.605794 19314 raft_consensus.cc:482] T 353ecf71ac51445087fa3f07e2836db5 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:16:23.605896 19314 raft_consensus.cc:3037] T 353ecf71ac51445087fa3f07e2836db5 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:23.607658 19314 raft_consensus.cc:504] T 353ecf71ac51445087fa3f07e2836db5 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:23.608301 19314 leader_election.cc:288] T 353ecf71ac51445087fa3f07e2836db5 P 4325d7d0ed484ad78a6ee54e9a003431 [CANDIDATE]: Term 1 election: Requested vote from peers 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837), 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647)
I20240503 02:16:23.608613 17763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "353ecf71ac51445087fa3f07e2836db5" candidate_uuid: "4325d7d0ed484ad78a6ee54e9a003431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836b43a7ee9146948319b8d37434b630"
I20240503 02:16:23.608685 17895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "353ecf71ac51445087fa3f07e2836db5" candidate_uuid: "4325d7d0ed484ad78a6ee54e9a003431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
I20240503 02:16:23.608814 17763 raft_consensus.cc:3037] T 353ecf71ac51445087fa3f07e2836db5 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:23.608877 17895 raft_consensus.cc:3037] T 353ecf71ac51445087fa3f07e2836db5 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:23.611652 17895 raft_consensus.cc:2454] T 353ecf71ac51445087fa3f07e2836db5 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4325d7d0ed484ad78a6ee54e9a003431 in term 1.
I20240503 02:16:23.611673 17763 raft_consensus.cc:2454] T 353ecf71ac51445087fa3f07e2836db5 P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4325d7d0ed484ad78a6ee54e9a003431 in term 1.
I20240503 02:16:23.612048 18088 leader_election.cc:302] T 353ecf71ac51445087fa3f07e2836db5 P 4325d7d0ed484ad78a6ee54e9a003431 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 4325d7d0ed484ad78a6ee54e9a003431; no voters:
I20240503 02:16:23.612267 19314 raft_consensus.cc:2781] T 353ecf71ac51445087fa3f07e2836db5 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:16:23.612414 19314 raft_consensus.cc:686] T 353ecf71ac51445087fa3f07e2836db5 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 LEADER]: Becoming Leader. State: Replica: 4325d7d0ed484ad78a6ee54e9a003431, State: Running, Role: LEADER
I20240503 02:16:23.612694 19314 consensus_queue.cc:237] T 353ecf71ac51445087fa3f07e2836db5 P 4325d7d0ed484ad78a6ee54e9a003431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:23.615593 17620 catalog_manager.cc:5484] T 353ecf71ac51445087fa3f07e2836db5 P 4325d7d0ed484ad78a6ee54e9a003431 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4). New cstate: current_term: 1 leader_uuid: "4325d7d0ed484ad78a6ee54e9a003431" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } }
I20240503 02:16:23.691704 19314 raft_consensus.cc:482] T 954f9a466d8648958df7baa916ea8558 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:23.691912 19314 raft_consensus.cc:504] T 954f9a466d8648958df7baa916ea8558 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:23.692667 19314 leader_election.cc:288] T 954f9a466d8648958df7baa916ea8558 P 4325d7d0ed484ad78a6ee54e9a003431 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759), 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837)
I20240503 02:16:23.693049 18025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "954f9a466d8648958df7baa916ea8558" candidate_uuid: "4325d7d0ed484ad78a6ee54e9a003431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd8517d70d94ae494e713df7eb61b12" is_pre_election: true
I20240503 02:16:23.693197 17763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "954f9a466d8648958df7baa916ea8558" candidate_uuid: "4325d7d0ed484ad78a6ee54e9a003431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836b43a7ee9146948319b8d37434b630" is_pre_election: true
I20240503 02:16:23.693287 18025 raft_consensus.cc:2454] T 954f9a466d8648958df7baa916ea8558 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4325d7d0ed484ad78a6ee54e9a003431 in term 0.
I20240503 02:16:23.693431 17763 raft_consensus.cc:2454] T 954f9a466d8648958df7baa916ea8558 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4325d7d0ed484ad78a6ee54e9a003431 in term 0.
I20240503 02:16:23.693745 18091 leader_election.cc:302] T 954f9a466d8648958df7baa916ea8558 P 4325d7d0ed484ad78a6ee54e9a003431 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cd8517d70d94ae494e713df7eb61b12, 4325d7d0ed484ad78a6ee54e9a003431; no voters:
I20240503 02:16:23.694031 19314 raft_consensus.cc:2781] T 954f9a466d8648958df7baa916ea8558 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:16:23.694128 19314 raft_consensus.cc:482] T 954f9a466d8648958df7baa916ea8558 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:16:23.694216 19314 raft_consensus.cc:3037] T 954f9a466d8648958df7baa916ea8558 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:23.696184 19314 raft_consensus.cc:504] T 954f9a466d8648958df7baa916ea8558 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:23.696715 19314 leader_election.cc:288] T 954f9a466d8648958df7baa916ea8558 P 4325d7d0ed484ad78a6ee54e9a003431 [CANDIDATE]: Term 1 election: Requested vote from peers 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759), 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837)
I20240503 02:16:23.697122 18025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "954f9a466d8648958df7baa916ea8558" candidate_uuid: "4325d7d0ed484ad78a6ee54e9a003431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:23.697201 17763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "954f9a466d8648958df7baa916ea8558" candidate_uuid: "4325d7d0ed484ad78a6ee54e9a003431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836b43a7ee9146948319b8d37434b630"
I20240503 02:16:23.697412 17763 raft_consensus.cc:3037] T 954f9a466d8648958df7baa916ea8558 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:23.697346 18025 raft_consensus.cc:3037] T 954f9a466d8648958df7baa916ea8558 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:23.699566 17763 raft_consensus.cc:2454] T 954f9a466d8648958df7baa916ea8558 P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4325d7d0ed484ad78a6ee54e9a003431 in term 1.
I20240503 02:16:23.699725 18025 raft_consensus.cc:2454] T 954f9a466d8648958df7baa916ea8558 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4325d7d0ed484ad78a6ee54e9a003431 in term 1.
I20240503 02:16:23.699918 18088 leader_election.cc:302] T 954f9a466d8648958df7baa916ea8558 P 4325d7d0ed484ad78a6ee54e9a003431 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4325d7d0ed484ad78a6ee54e9a003431, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:23.700179 19314 raft_consensus.cc:2781] T 954f9a466d8648958df7baa916ea8558 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:16:23.700310 19314 raft_consensus.cc:686] T 954f9a466d8648958df7baa916ea8558 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 LEADER]: Becoming Leader. State: Replica: 4325d7d0ed484ad78a6ee54e9a003431, State: Running, Role: LEADER
I20240503 02:16:23.700582 19314 consensus_queue.cc:237] T 954f9a466d8648958df7baa916ea8558 P 4325d7d0ed484ad78a6ee54e9a003431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:23.703312 17620 catalog_manager.cc:5484] T 954f9a466d8648958df7baa916ea8558 P 4325d7d0ed484ad78a6ee54e9a003431 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4). New cstate: current_term: 1 leader_uuid: "4325d7d0ed484ad78a6ee54e9a003431" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } }
I20240503 02:16:23.960049 19128 consensus_queue.cc:1035] T bb6aa5daccb6436fb7910d2d28244b7c P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:16:23.965253 19451 consensus_queue.cc:1035] T bb6aa5daccb6436fb7910d2d28244b7c P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:16:24.070571 19456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:16:24.070896 19456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 02:16:24.074889 19314 consensus_queue.cc:1035] T 353ecf71ac51445087fa3f07e2836db5 P 4325d7d0ed484ad78a6ee54e9a003431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:16:24.081018 19456 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:16:24.081423 19314 consensus_queue.cc:1035] T 353ecf71ac51445087fa3f07e2836db5 P 4325d7d0ed484ad78a6ee54e9a003431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_8'
I20240503 02:16:24.223949 17763 raft_consensus.cc:1261] T 954f9a466d8648958df7baa916ea8558 P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Refusing update from remote peer 4325d7d0ed484ad78a6ee54e9a003431: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:16:24.224862 18025 raft_consensus.cc:1261] T 954f9a466d8648958df7baa916ea8558 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Refusing update from remote peer 4325d7d0ed484ad78a6ee54e9a003431: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:16:24.226372 19384 consensus_queue.cc:1035] T 954f9a466d8648958df7baa916ea8558 P 4325d7d0ed484ad78a6ee54e9a003431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:16:24.245795 19496 consensus_queue.cc:1035] T 954f9a466d8648958df7baa916ea8558 P 4325d7d0ed484ad78a6ee54e9a003431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 230.171 ms
time per row: 0.115086 ms
I20240503 02:16:24.470309 17619 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44922:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:16:24.480250 17873 tablet_service.cc:1467] Processing CreateTablet for tablet c0300c0a94654f08b08f0ef0cea1d6be (DEFAULT_TABLE table=rebalance_test_table_9 [id=f0037053741747cea25ff511ccf10dec]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:24.480360 18133 tablet_service.cc:1467] Processing CreateTablet for tablet c0300c0a94654f08b08f0ef0cea1d6be (DEFAULT_TABLE table=rebalance_test_table_9 [id=f0037053741747cea25ff511ccf10dec]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:24.480769 17873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0300c0a94654f08b08f0ef0cea1d6be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:24.480769 18133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0300c0a94654f08b08f0ef0cea1d6be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:24.481134 18004 tablet_service.cc:1467] Processing CreateTablet for tablet c0300c0a94654f08b08f0ef0cea1d6be (DEFAULT_TABLE table=rebalance_test_table_9 [id=f0037053741747cea25ff511ccf10dec]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:24.481528 18004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0300c0a94654f08b08f0ef0cea1d6be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:24.481513 18135 tablet_service.cc:1467] Processing CreateTablet for tablet 736e23a32ac944e2825262be0f7d3992 (DEFAULT_TABLE table=rebalance_test_table_9 [id=f0037053741747cea25ff511ccf10dec]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:24.481874 18135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 736e23a32ac944e2825262be0f7d3992. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:24.486328 18005 tablet_service.cc:1467] Processing CreateTablet for tablet 736e23a32ac944e2825262be0f7d3992 (DEFAULT_TABLE table=rebalance_test_table_9 [id=f0037053741747cea25ff511ccf10dec]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:24.486428 18003 tablet_service.cc:1467] Processing CreateTablet for tablet 0fda016a9d3b4f238c0ed4bcf47a00b1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=f0037053741747cea25ff511ccf10dec]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:24.486690 18005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 736e23a32ac944e2825262be0f7d3992. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:24.487735 17743 tablet_service.cc:1467] Processing CreateTablet for tablet 0fda016a9d3b4f238c0ed4bcf47a00b1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=f0037053741747cea25ff511ccf10dec]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:24.488197 17743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fda016a9d3b4f238c0ed4bcf47a00b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:24.490162 17874 tablet_service.cc:1467] Processing CreateTablet for tablet 736e23a32ac944e2825262be0f7d3992 (DEFAULT_TABLE table=rebalance_test_table_9 [id=f0037053741747cea25ff511ccf10dec]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:24.490511 17874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 736e23a32ac944e2825262be0f7d3992. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:24.494693 17875 tablet_service.cc:1467] Processing CreateTablet for tablet 0fda016a9d3b4f238c0ed4bcf47a00b1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=f0037053741747cea25ff511ccf10dec]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:16:24.495093 17875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fda016a9d3b4f238c0ed4bcf47a00b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:24.495162 18003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fda016a9d3b4f238c0ed4bcf47a00b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:24.503756 19518 tablet_bootstrap.cc:492] T 736e23a32ac944e2825262be0f7d3992 P 4325d7d0ed484ad78a6ee54e9a003431: Bootstrap starting.
I20240503 02:16:24.505097 19519 tablet_bootstrap.cc:492] T 736e23a32ac944e2825262be0f7d3992 P 1cd8517d70d94ae494e713df7eb61b12: Bootstrap starting.
I20240503 02:16:24.505944 19518 tablet_bootstrap.cc:654] T 736e23a32ac944e2825262be0f7d3992 P 4325d7d0ed484ad78a6ee54e9a003431: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:24.507025 19519 tablet_bootstrap.cc:654] T 736e23a32ac944e2825262be0f7d3992 P 1cd8517d70d94ae494e713df7eb61b12: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:24.509819 19517 tablet_bootstrap.cc:492] T c0300c0a94654f08b08f0ef0cea1d6be P 0f3838ee7ad04e94aa65ac750dad0833: Bootstrap starting.
I20240503 02:16:24.512398 19517 tablet_bootstrap.cc:654] T c0300c0a94654f08b08f0ef0cea1d6be P 0f3838ee7ad04e94aa65ac750dad0833: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:24.514139 19522 tablet_bootstrap.cc:492] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 836b43a7ee9146948319b8d37434b630: Bootstrap starting.
I20240503 02:16:24.517207 19522 tablet_bootstrap.cc:654] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 836b43a7ee9146948319b8d37434b630: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:24.518050 19519 tablet_bootstrap.cc:492] T 736e23a32ac944e2825262be0f7d3992 P 1cd8517d70d94ae494e713df7eb61b12: No bootstrap required, opened a new log
I20240503 02:16:24.518298 19519 ts_tablet_manager.cc:1367] T 736e23a32ac944e2825262be0f7d3992 P 1cd8517d70d94ae494e713df7eb61b12: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20240503 02:16:24.519325 19519 raft_consensus.cc:348] T 736e23a32ac944e2825262be0f7d3992 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:24.519667 19519 raft_consensus.cc:374] T 736e23a32ac944e2825262be0f7d3992 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:24.519810 19519 raft_consensus.cc:729] T 736e23a32ac944e2825262be0f7d3992 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Initialized, Role: FOLLOWER
I20240503 02:16:24.519949 19522 tablet_bootstrap.cc:492] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 836b43a7ee9146948319b8d37434b630: No bootstrap required, opened a new log
I20240503 02:16:24.520044 19518 tablet_bootstrap.cc:492] T 736e23a32ac944e2825262be0f7d3992 P 4325d7d0ed484ad78a6ee54e9a003431: No bootstrap required, opened a new log
I20240503 02:16:24.520134 19522 ts_tablet_manager.cc:1367] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 836b43a7ee9146948319b8d37434b630: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20240503 02:16:24.520150 19519 consensus_queue.cc:260] T 736e23a32ac944e2825262be0f7d3992 P 1cd8517d70d94ae494e713df7eb61b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:24.520278 19518 ts_tablet_manager.cc:1367] T 736e23a32ac944e2825262be0f7d3992 P 4325d7d0ed484ad78a6ee54e9a003431: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.012s
I20240503 02:16:24.520932 19519 ts_tablet_manager.cc:1398] T 736e23a32ac944e2825262be0f7d3992 P 1cd8517d70d94ae494e713df7eb61b12: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:16:24.520943 19522 raft_consensus.cc:348] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:24.521239 19522 raft_consensus.cc:374] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:24.521306 19519 tablet_bootstrap.cc:492] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 1cd8517d70d94ae494e713df7eb61b12: Bootstrap starting.
I20240503 02:16:24.521201 19518 raft_consensus.cc:348] T 736e23a32ac944e2825262be0f7d3992 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:24.521384 19522 raft_consensus.cc:729] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Initialized, Role: FOLLOWER
I20240503 02:16:24.521390 19518 raft_consensus.cc:374] T 736e23a32ac944e2825262be0f7d3992 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:24.521492 19518 raft_consensus.cc:729] T 736e23a32ac944e2825262be0f7d3992 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4325d7d0ed484ad78a6ee54e9a003431, State: Initialized, Role: FOLLOWER
I20240503 02:16:24.521690 19522 consensus_queue.cc:260] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 836b43a7ee9146948319b8d37434b630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:24.521785 19518 consensus_queue.cc:260] T 736e23a32ac944e2825262be0f7d3992 P 4325d7d0ed484ad78a6ee54e9a003431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:24.522459 19518 ts_tablet_manager.cc:1398] T 736e23a32ac944e2825262be0f7d3992 P 4325d7d0ed484ad78a6ee54e9a003431: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:16:24.522748 19518 tablet_bootstrap.cc:492] T c0300c0a94654f08b08f0ef0cea1d6be P 4325d7d0ed484ad78a6ee54e9a003431: Bootstrap starting.
I20240503 02:16:24.523262 19522 ts_tablet_manager.cc:1398] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 836b43a7ee9146948319b8d37434b630: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:16:24.524509 19519 tablet_bootstrap.cc:654] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 1cd8517d70d94ae494e713df7eb61b12: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:24.524787 19518 tablet_bootstrap.cc:654] T c0300c0a94654f08b08f0ef0cea1d6be P 4325d7d0ed484ad78a6ee54e9a003431: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:24.528510 19519 tablet_bootstrap.cc:492] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 1cd8517d70d94ae494e713df7eb61b12: No bootstrap required, opened a new log
I20240503 02:16:24.528717 19519 ts_tablet_manager.cc:1367] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 1cd8517d70d94ae494e713df7eb61b12: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240503 02:16:24.529542 19519 raft_consensus.cc:348] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:24.529867 19519 raft_consensus.cc:374] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:24.530000 19519 raft_consensus.cc:729] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Initialized, Role: FOLLOWER
I20240503 02:16:24.530309 19519 consensus_queue.cc:260] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 1cd8517d70d94ae494e713df7eb61b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:24.530992 19519 ts_tablet_manager.cc:1398] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 1cd8517d70d94ae494e713df7eb61b12: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:16:24.531755 19519 tablet_bootstrap.cc:492] T c0300c0a94654f08b08f0ef0cea1d6be P 1cd8517d70d94ae494e713df7eb61b12: Bootstrap starting.
I20240503 02:16:24.533428 19519 tablet_bootstrap.cc:654] T c0300c0a94654f08b08f0ef0cea1d6be P 1cd8517d70d94ae494e713df7eb61b12: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:24.534368 19518 tablet_bootstrap.cc:492] T c0300c0a94654f08b08f0ef0cea1d6be P 4325d7d0ed484ad78a6ee54e9a003431: No bootstrap required, opened a new log
I20240503 02:16:24.534582 19518 ts_tablet_manager.cc:1367] T c0300c0a94654f08b08f0ef0cea1d6be P 4325d7d0ed484ad78a6ee54e9a003431: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20240503 02:16:24.534663 19517 tablet_bootstrap.cc:492] T c0300c0a94654f08b08f0ef0cea1d6be P 0f3838ee7ad04e94aa65ac750dad0833: No bootstrap required, opened a new log
I20240503 02:16:24.534916 19517 ts_tablet_manager.cc:1367] T c0300c0a94654f08b08f0ef0cea1d6be P 0f3838ee7ad04e94aa65ac750dad0833: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.000s
I20240503 02:16:24.535707 19518 raft_consensus.cc:348] T c0300c0a94654f08b08f0ef0cea1d6be P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:24.536020 19518 raft_consensus.cc:374] T c0300c0a94654f08b08f0ef0cea1d6be P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:24.536145 19518 raft_consensus.cc:729] T c0300c0a94654f08b08f0ef0cea1d6be P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4325d7d0ed484ad78a6ee54e9a003431, State: Initialized, Role: FOLLOWER
I20240503 02:16:24.536079 19517 raft_consensus.cc:348] T c0300c0a94654f08b08f0ef0cea1d6be P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:24.536465 19517 raft_consensus.cc:374] T c0300c0a94654f08b08f0ef0cea1d6be P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:24.536439 19518 consensus_queue.cc:260] T c0300c0a94654f08b08f0ef0cea1d6be P 4325d7d0ed484ad78a6ee54e9a003431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:24.536684 19517 raft_consensus.cc:729] T c0300c0a94654f08b08f0ef0cea1d6be P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Initialized, Role: FOLLOWER
I20240503 02:16:24.537050 19517 consensus_queue.cc:260] T c0300c0a94654f08b08f0ef0cea1d6be P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:24.537307 19518 ts_tablet_manager.cc:1398] T c0300c0a94654f08b08f0ef0cea1d6be P 4325d7d0ed484ad78a6ee54e9a003431: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 02:16:24.540733 19517 ts_tablet_manager.cc:1398] T c0300c0a94654f08b08f0ef0cea1d6be P 0f3838ee7ad04e94aa65ac750dad0833: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 02:16:24.541805 19517 tablet_bootstrap.cc:492] T 736e23a32ac944e2825262be0f7d3992 P 0f3838ee7ad04e94aa65ac750dad0833: Bootstrap starting.
I20240503 02:16:24.542409 19519 tablet_bootstrap.cc:492] T c0300c0a94654f08b08f0ef0cea1d6be P 1cd8517d70d94ae494e713df7eb61b12: No bootstrap required, opened a new log
I20240503 02:16:24.542524 19519 ts_tablet_manager.cc:1367] T c0300c0a94654f08b08f0ef0cea1d6be P 1cd8517d70d94ae494e713df7eb61b12: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20240503 02:16:24.543057 19519 raft_consensus.cc:348] T c0300c0a94654f08b08f0ef0cea1d6be P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:24.543247 19519 raft_consensus.cc:374] T c0300c0a94654f08b08f0ef0cea1d6be P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:24.543347 19519 raft_consensus.cc:729] T c0300c0a94654f08b08f0ef0cea1d6be P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Initialized, Role: FOLLOWER
I20240503 02:16:24.543402 19517 tablet_bootstrap.cc:654] T 736e23a32ac944e2825262be0f7d3992 P 0f3838ee7ad04e94aa65ac750dad0833: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:24.543545 19519 consensus_queue.cc:260] T c0300c0a94654f08b08f0ef0cea1d6be P 1cd8517d70d94ae494e713df7eb61b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:24.544698 19519 ts_tablet_manager.cc:1398] T c0300c0a94654f08b08f0ef0cea1d6be P 1cd8517d70d94ae494e713df7eb61b12: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:16:24.545218 19517 tablet_bootstrap.cc:492] T 736e23a32ac944e2825262be0f7d3992 P 0f3838ee7ad04e94aa65ac750dad0833: No bootstrap required, opened a new log
I20240503 02:16:24.545375 19517 ts_tablet_manager.cc:1367] T 736e23a32ac944e2825262be0f7d3992 P 0f3838ee7ad04e94aa65ac750dad0833: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 02:16:24.546026 19517 raft_consensus.cc:348] T 736e23a32ac944e2825262be0f7d3992 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:24.546252 19517 raft_consensus.cc:374] T 736e23a32ac944e2825262be0f7d3992 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:24.546330 19517 raft_consensus.cc:729] T 736e23a32ac944e2825262be0f7d3992 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Initialized, Role: FOLLOWER
I20240503 02:16:24.546582 19517 consensus_queue.cc:260] T 736e23a32ac944e2825262be0f7d3992 P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:24.547220 19517 ts_tablet_manager.cc:1398] T 736e23a32ac944e2825262be0f7d3992 P 0f3838ee7ad04e94aa65ac750dad0833: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:16:24.547538 19517 tablet_bootstrap.cc:492] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 0f3838ee7ad04e94aa65ac750dad0833: Bootstrap starting.
I20240503 02:16:24.548776 19517 tablet_bootstrap.cc:654] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 0f3838ee7ad04e94aa65ac750dad0833: Neither blocks nor log segments found. Creating new log.
I20240503 02:16:24.550593 19517 tablet_bootstrap.cc:492] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 0f3838ee7ad04e94aa65ac750dad0833: No bootstrap required, opened a new log
I20240503 02:16:24.550706 19517 ts_tablet_manager.cc:1367] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 0f3838ee7ad04e94aa65ac750dad0833: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:16:24.551360 19517 raft_consensus.cc:348] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:24.551561 19517 raft_consensus.cc:374] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:16:24.551651 19517 raft_consensus.cc:729] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Initialized, Role: FOLLOWER
I20240503 02:16:24.551851 19517 consensus_queue.cc:260] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:24.552330 19517 ts_tablet_manager.cc:1398] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 0f3838ee7ad04e94aa65ac750dad0833: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:16:24.590332 19454 raft_consensus.cc:482] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:24.590494 19454 raft_consensus.cc:504] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:24.591137 19454 leader_election.cc:288] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837), 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647)
I20240503 02:16:24.591600 17763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fda016a9d3b4f238c0ed4bcf47a00b1" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836b43a7ee9146948319b8d37434b630" is_pre_election: true
I20240503 02:16:24.591631 17895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fda016a9d3b4f238c0ed4bcf47a00b1" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833" is_pre_election: true
I20240503 02:16:24.591827 17895 raft_consensus.cc:2454] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cd8517d70d94ae494e713df7eb61b12 in term 0.
I20240503 02:16:24.591821 17763 raft_consensus.cc:2454] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cd8517d70d94ae494e713df7eb61b12 in term 0.
I20240503 02:16:24.592221 17958 leader_election.cc:302] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 1cd8517d70d94ae494e713df7eb61b12; no voters:
I20240503 02:16:24.592458 19454 raft_consensus.cc:2781] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:16:24.592566 19454 raft_consensus.cc:482] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:16:24.592671 19454 raft_consensus.cc:3037] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:24.594753 19454 raft_consensus.cc:504] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:24.595235 19454 leader_election.cc:288] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 1 election: Requested vote from peers 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837), 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647)
I20240503 02:16:24.595613 17763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fda016a9d3b4f238c0ed4bcf47a00b1" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836b43a7ee9146948319b8d37434b630"
I20240503 02:16:24.595656 17895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fda016a9d3b4f238c0ed4bcf47a00b1" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
I20240503 02:16:24.595808 17895 raft_consensus.cc:3037] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:24.595805 17763 raft_consensus.cc:3037] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 836b43a7ee9146948319b8d37434b630 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:24.597685 17895 raft_consensus.cc:2454] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cd8517d70d94ae494e713df7eb61b12 in term 1.
I20240503 02:16:24.597703 17763 raft_consensus.cc:2454] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cd8517d70d94ae494e713df7eb61b12 in term 1.
I20240503 02:16:24.598062 17958 leader_election.cc:302] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 1cd8517d70d94ae494e713df7eb61b12; no voters:
I20240503 02:16:24.598290 19454 raft_consensus.cc:2781] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:16:24.598680 19454 raft_consensus.cc:686] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 LEADER]: Becoming Leader. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Running, Role: LEADER
I20240503 02:16:24.599009 19454 consensus_queue.cc:237] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 1cd8517d70d94ae494e713df7eb61b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:24.601852 17621 catalog_manager.cc:5484] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 1cd8517d70d94ae494e713df7eb61b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3). New cstate: current_term: 1 leader_uuid: "1cd8517d70d94ae494e713df7eb61b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } }
I20240503 02:16:24.799027 19512 raft_consensus.cc:482] T 736e23a32ac944e2825262be0f7d3992 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:24.799202 19512 raft_consensus.cc:504] T 736e23a32ac944e2825262be0f7d3992 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:24.799798 19512 leader_election.cc:288] T 736e23a32ac944e2825262be0f7d3992 P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703), 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759)
I20240503 02:16:24.800143 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736e23a32ac944e2825262be0f7d3992" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431" is_pre_election: true
I20240503 02:16:24.800356 18155 raft_consensus.cc:2454] T 736e23a32ac944e2825262be0f7d3992 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 0.
I20240503 02:16:24.800343 18024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736e23a32ac944e2825262be0f7d3992" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd8517d70d94ae494e713df7eb61b12" is_pre_election: true
I20240503 02:16:24.800581 18024 raft_consensus.cc:2454] T 736e23a32ac944e2825262be0f7d3992 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 0.
I20240503 02:16:24.800773 17828 leader_election.cc:302] T 736e23a32ac944e2825262be0f7d3992 P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 4325d7d0ed484ad78a6ee54e9a003431; no voters:
I20240503 02:16:24.801051 19512 raft_consensus.cc:2781] T 736e23a32ac944e2825262be0f7d3992 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:16:24.801174 19512 raft_consensus.cc:482] T 736e23a32ac944e2825262be0f7d3992 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:16:24.801259 19512 raft_consensus.cc:3037] T 736e23a32ac944e2825262be0f7d3992 P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:24.803432 19512 raft_consensus.cc:504] T 736e23a32ac944e2825262be0f7d3992 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:24.803882 19512 leader_election.cc:288] T 736e23a32ac944e2825262be0f7d3992 P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 1 election: Requested vote from peers 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703), 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759)
I20240503 02:16:24.804190 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736e23a32ac944e2825262be0f7d3992" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431"
I20240503 02:16:24.804371 18155 raft_consensus.cc:3037] T 736e23a32ac944e2825262be0f7d3992 P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:24.804412 18024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736e23a32ac944e2825262be0f7d3992" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:24.804607 18024 raft_consensus.cc:3037] T 736e23a32ac944e2825262be0f7d3992 P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:24.807160 18155 raft_consensus.cc:2454] T 736e23a32ac944e2825262be0f7d3992 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 1.
I20240503 02:16:24.807263 18024 raft_consensus.cc:2454] T 736e23a32ac944e2825262be0f7d3992 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 1.
I20240503 02:16:24.807541 17828 leader_election.cc:302] T 736e23a32ac944e2825262be0f7d3992 P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 4325d7d0ed484ad78a6ee54e9a003431; no voters:
I20240503 02:16:24.807883 19512 raft_consensus.cc:2781] T 736e23a32ac944e2825262be0f7d3992 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:16:24.808022 19512 raft_consensus.cc:686] T 736e23a32ac944e2825262be0f7d3992 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 LEADER]: Becoming Leader. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:24.808279 19512 consensus_queue.cc:237] T 736e23a32ac944e2825262be0f7d3992 P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:24.811022 17621 catalog_manager.cc:5484] T 736e23a32ac944e2825262be0f7d3992 P 0f3838ee7ad04e94aa65ac750dad0833 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2). New cstate: current_term: 1 leader_uuid: "0f3838ee7ad04e94aa65ac750dad0833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } }
I20240503 02:16:24.851176 19496 raft_consensus.cc:482] T c0300c0a94654f08b08f0ef0cea1d6be P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:16:24.851395 19496 raft_consensus.cc:504] T c0300c0a94654f08b08f0ef0cea1d6be P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:24.852190 19496 leader_election.cc:288] T c0300c0a94654f08b08f0ef0cea1d6be P 4325d7d0ed484ad78a6ee54e9a003431 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759), 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647)
I20240503 02:16:24.852731 17895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0300c0a94654f08b08f0ef0cea1d6be" candidate_uuid: "4325d7d0ed484ad78a6ee54e9a003431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833" is_pre_election: true
I20240503 02:16:24.853003 17895 raft_consensus.cc:2454] T c0300c0a94654f08b08f0ef0cea1d6be P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4325d7d0ed484ad78a6ee54e9a003431 in term 0.
I20240503 02:16:24.852958 18024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0300c0a94654f08b08f0ef0cea1d6be" candidate_uuid: "4325d7d0ed484ad78a6ee54e9a003431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd8517d70d94ae494e713df7eb61b12" is_pre_election: true
I20240503 02:16:24.853191 18024 raft_consensus.cc:2454] T c0300c0a94654f08b08f0ef0cea1d6be P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4325d7d0ed484ad78a6ee54e9a003431 in term 0.
I20240503 02:16:24.853420 18088 leader_election.cc:302] T c0300c0a94654f08b08f0ef0cea1d6be P 4325d7d0ed484ad78a6ee54e9a003431 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 4325d7d0ed484ad78a6ee54e9a003431; no voters:
I20240503 02:16:24.853684 19496 raft_consensus.cc:2781] T c0300c0a94654f08b08f0ef0cea1d6be P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:16:24.853828 19496 raft_consensus.cc:482] T c0300c0a94654f08b08f0ef0cea1d6be P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:16:24.853984 19496 raft_consensus.cc:3037] T c0300c0a94654f08b08f0ef0cea1d6be P 4325d7d0ed484ad78a6ee54e9a003431 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:24.856019 19496 raft_consensus.cc:504] T c0300c0a94654f08b08f0ef0cea1d6be P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:24.856709 19496 leader_election.cc:288] T c0300c0a94654f08b08f0ef0cea1d6be P 4325d7d0ed484ad78a6ee54e9a003431 [CANDIDATE]: Term 1 election: Requested vote from peers 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759), 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647)
I20240503 02:16:24.857084 18024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0300c0a94654f08b08f0ef0cea1d6be" candidate_uuid: "4325d7d0ed484ad78a6ee54e9a003431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:24.857205 17895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0300c0a94654f08b08f0ef0cea1d6be" candidate_uuid: "4325d7d0ed484ad78a6ee54e9a003431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
I20240503 02:16:24.857298 18024 raft_consensus.cc:3037] T c0300c0a94654f08b08f0ef0cea1d6be P 1cd8517d70d94ae494e713df7eb61b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:24.857427 17895 raft_consensus.cc:3037] T c0300c0a94654f08b08f0ef0cea1d6be P 0f3838ee7ad04e94aa65ac750dad0833 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:16:24.859220 17895 raft_consensus.cc:2454] T c0300c0a94654f08b08f0ef0cea1d6be P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4325d7d0ed484ad78a6ee54e9a003431 in term 1.
I20240503 02:16:24.859958 18088 leader_election.cc:302] T c0300c0a94654f08b08f0ef0cea1d6be P 4325d7d0ed484ad78a6ee54e9a003431 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 4325d7d0ed484ad78a6ee54e9a003431; no voters:
I20240503 02:16:24.859966 18024 raft_consensus.cc:2454] T c0300c0a94654f08b08f0ef0cea1d6be P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4325d7d0ed484ad78a6ee54e9a003431 in term 1.
I20240503 02:16:24.860199 19496 raft_consensus.cc:2781] T c0300c0a94654f08b08f0ef0cea1d6be P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:16:24.860644 19496 raft_consensus.cc:686] T c0300c0a94654f08b08f0ef0cea1d6be P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 LEADER]: Becoming Leader. State: Replica: 4325d7d0ed484ad78a6ee54e9a003431, State: Running, Role: LEADER
I20240503 02:16:24.861140 19496 consensus_queue.cc:237] T c0300c0a94654f08b08f0ef0cea1d6be P 4325d7d0ed484ad78a6ee54e9a003431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:24.864382 17621 catalog_manager.cc:5484] T c0300c0a94654f08b08f0ef0cea1d6be P 4325d7d0ed484ad78a6ee54e9a003431 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4). New cstate: current_term: 1 leader_uuid: "4325d7d0ed484ad78a6ee54e9a003431" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } }
I20240503 02:16:25.193585 19454 consensus_queue.cc:1035] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 1cd8517d70d94ae494e713df7eb61b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:16:25.198431 19512 consensus_queue.cc:1035] T 736e23a32ac944e2825262be0f7d3992 P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:16:25.210451 19454 consensus_queue.cc:1035] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 1cd8517d70d94ae494e713df7eb61b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:16:25.222365 19547 consensus_queue.cc:1035] T 736e23a32ac944e2825262be0f7d3992 P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240503 02:16:25.239408 19535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:16:25.239899 19535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:16:25.247517 19535 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:16:25.259788 19496 consensus_queue.cc:1035] T c0300c0a94654f08b08f0ef0cea1d6be P 4325d7d0ed484ad78a6ee54e9a003431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:16:25.264540 19533 consensus_queue.cc:1035] T c0300c0a94654f08b08f0ef0cea1d6be P 4325d7d0ed484ad78a6ee54e9a003431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_9'
INSERT report
rows total: 2000
time total: 240.394 ms
time per row: 0.120197 ms
I20240503 02:16:25.635083 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.5:37187
--local_ip_for_outbound_sockets=127.22.108.5
--tserver_master_addrs=127.22.108.62:34029
--webserver_port=41917
--webserver_interface=127.22.108.5
--builtin_ntp_servers=127.22.108.20:46405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 02:16:25.994756 19599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:16:25.995040 19599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:16:25.995178 19599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:16:26.002915 19599 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:16:26.003044 19599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:16:26.003212 19599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.5
I20240503 02:16:26.013597 19599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:46405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.5:37187
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.22.108.5
--webserver_port=41917
--tserver_master_addrs=127.22.108.62:34029
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.5
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:16:26.015053 19599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:16:26.016963 19599 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:16:26.029829 19606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:16:26.030560 19605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:16:26.031414 19608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:16:26.032251 19599 server_base.cc:1034] running on GCE node
I20240503 02:16:26.032909 19599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:16:26.034097 19599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:16:26.035317 19599 hybrid_clock.cc:648] HybridClock initialized: now 1714702586035250 us; error 61 us; skew 500 ppm
I20240503 02:16:26.035670 19599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:16:26.038137 19599 webserver.cc:460] Webserver started at http://127.22.108.5:41917/ using document root <none> and password file <none>
I20240503 02:16:26.038877 19599 fs_manager.cc:359] Metadata directory not provided
I20240503 02:16:26.039016 19599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:16:26.044425 19599 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20240503 02:16:26.047194 19614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:16:26.048519 19599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 02:16:26.048678 19599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "7a727cc21a924056b804ff5460b06d9a"
format_stamp: "Formatted at 2024-05-03 02:16:07 on dist-test-slave-sc15"
I20240503 02:16:26.049513 19599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:16:26.090430 19599 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:16:26.091327 19599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:16:26.091619 19599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:16:26.092607 19599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:16:26.094331 19599 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:16:26.094414 19599 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:16:26.094552 19599 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:16:26.094605 19599 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:16:26.132663 19599 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.5:37187
I20240503 02:16:26.132740 19727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.5:37187 every 8 connection(s)
I20240503 02:16:26.134212 19599 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:16:26.141520 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 19599
I20240503 02:16:26.142243 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.6:43647
--local_ip_for_outbound_sockets=127.22.108.6
--tserver_master_addrs=127.22.108.62:34029
--webserver_port=45105
--webserver_interface=127.22.108.6
--builtin_ntp_servers=127.22.108.20:46405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:16:26.147367 19728 heartbeater.cc:347] Connected to a master server at 127.22.108.62:34029
I20240503 02:16:26.147645 19728 heartbeater.cc:464] Registering TS with master...
I20240503 02:16:26.148406 19728 heartbeater.cc:511] Master 127.22.108.62:34029 requested a full tablet report, sending...
I20240503 02:16:26.151556 17621 ts_manager.cc:194] Re-registered known tserver with Master: 7a727cc21a924056b804ff5460b06d9a (127.22.108.5:37187)
I20240503 02:16:26.152350 17621 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.5:53675
W20240503 02:16:26.484246 19732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:16:26.484520 19732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:16:26.484637 19732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:16:26.492323 19732 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:16:26.492467 19732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:16:26.492678 19732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.6
I20240503 02:16:26.502493 19732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:46405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.6:43647
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.22.108.6
--webserver_port=45105
--tserver_master_addrs=127.22.108.62:34029
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.6
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:16:26.503795 19732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:16:26.505645 19732 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:16:26.518947 19737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:16:26.520023 19740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:16:26.520732 19738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:16:26.521654 19732 server_base.cc:1034] running on GCE node
I20240503 02:16:26.522397 19732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:16:26.523438 19732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:16:26.524646 19732 hybrid_clock.cc:648] HybridClock initialized: now 1714702586524611 us; error 46 us; skew 500 ppm
I20240503 02:16:26.524995 19732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:16:26.527652 19732 webserver.cc:460] Webserver started at http://127.22.108.6:45105/ using document root <none> and password file <none>
I20240503 02:16:26.528432 19732 fs_manager.cc:359] Metadata directory not provided
I20240503 02:16:26.528571 19732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:16:26.533608 19732 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:16:26.536573 19746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:16:26.538138 19732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:16:26.538290 19732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "314f3c1b901a4cfdb0ed6298b9b874f2"
format_stamp: "Formatted at 2024-05-03 02:16:07 on dist-test-slave-sc15"
I20240503 02:16:26.539075 19732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:16:26.568774 19732 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:16:26.569665 19732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:16:26.569999 19732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:16:26.570976 19732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:16:26.572422 19732 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:16:26.572490 19732 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:16:26.572605 19732 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:16:26.572652 19732 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:16:26.603266 19732 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.6:43647
I20240503 02:16:26.603318 19858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.6:43647 every 8 connection(s)
I20240503 02:16:26.604698 19732 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 02:16:26.607434 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 19732
I20240503 02:16:26.608002 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.7:35849
--local_ip_for_outbound_sockets=127.22.108.7
--tserver_master_addrs=127.22.108.62:34029
--webserver_port=41063
--webserver_interface=127.22.108.7
--builtin_ntp_servers=127.22.108.20:46405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:16:26.618543 19859 heartbeater.cc:347] Connected to a master server at 127.22.108.62:34029
I20240503 02:16:26.618852 19859 heartbeater.cc:464] Registering TS with master...
I20240503 02:16:26.619535 19859 heartbeater.cc:511] Master 127.22.108.62:34029 requested a full tablet report, sending...
I20240503 02:16:26.621109 17616 ts_manager.cc:194] Re-registered known tserver with Master: 314f3c1b901a4cfdb0ed6298b9b874f2 (127.22.108.6:43647)
I20240503 02:16:26.622289 17616 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.6:49091
W20240503 02:16:26.951416 19862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:16:26.951678 19862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:16:26.951768 19862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:16:26.959511 19862 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:16:26.959635 19862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:16:26.959751 19862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.7
I20240503 02:16:26.968596 19862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:46405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.7:35849
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.22.108.7
--webserver_port=41063
--tserver_master_addrs=127.22.108.62:34029
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.7
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:16:26.969935 19862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:16:26.971561 19862 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:16:26.983850 19867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:16:26.984499 19870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:16:26.985287 19868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:16:26.986531 19862 server_base.cc:1034] running on GCE node
I20240503 02:16:26.987280 19862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:16:26.988191 19862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:16:26.989382 19862 hybrid_clock.cc:648] HybridClock initialized: now 1714702586989330 us; error 40 us; skew 500 ppm
I20240503 02:16:26.989710 19862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:16:26.992087 19862 webserver.cc:460] Webserver started at http://127.22.108.7:41063/ using document root <none> and password file <none>
I20240503 02:16:26.992933 19862 fs_manager.cc:359] Metadata directory not provided
I20240503 02:16:26.993072 19862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:16:26.998159 19862 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20240503 02:16:27.000617 19876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:16:27.001889 19862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240503 02:16:27.002048 19862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "eceaab49d4184b68a38415bd66b8641a"
format_stamp: "Formatted at 2024-05-03 02:16:08 on dist-test-slave-sc15"
I20240503 02:16:27.002791 19862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:16:27.021504 19862 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:16:27.022313 19862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:16:27.022557 19862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:16:27.023495 19862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:16:27.025079 19862 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:16:27.025171 19862 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:16:27.025267 19862 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:16:27.025310 19862 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:16:27.056433 19862 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.7:35849
I20240503 02:16:27.056469 19988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.7:35849 every 8 connection(s)
I20240503 02:16:27.057933 19862 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 02:16:27.063535 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 19862
I20240503 02:16:27.064225 22960 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1Onrea/build/asan/bin/kudu
/tmp/dist-test-task1Onrea/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.108.8:32871
--local_ip_for_outbound_sockets=127.22.108.8
--tserver_master_addrs=127.22.108.62:34029
--webserver_port=41063
--webserver_interface=127.22.108.8
--builtin_ntp_servers=127.22.108.20:46405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:16:27.072149 19989 heartbeater.cc:347] Connected to a master server at 127.22.108.62:34029
I20240503 02:16:27.072484 19989 heartbeater.cc:464] Registering TS with master...
I20240503 02:16:27.073443 19989 heartbeater.cc:511] Master 127.22.108.62:34029 requested a full tablet report, sending...
I20240503 02:16:27.074766 17616 ts_manager.cc:194] Re-registered known tserver with Master: eceaab49d4184b68a38415bd66b8641a (127.22.108.7:35849)
I20240503 02:16:27.075913 17616 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.7:45765
I20240503 02:16:27.155019 19728 heartbeater.cc:503] Master 127.22.108.62:34029 was elected leader, sending a full tablet report...
W20240503 02:16:27.415536 19992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:16:27.415838 19992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:16:27.415930 19992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:16:27.423450 19992 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:16:27.423578 19992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:16:27.423696 19992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.108.8
I20240503 02:16:27.433593 19992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.108.20:46405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.108.8:32871
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.22.108.8
--webserver_port=41063
--tserver_master_addrs=127.22.108.62:34029
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.108.8
--log_dir=/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:16:27.434945 19992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:16:27.436959 19992 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:16:27.449755 19998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:16:27.450634 19999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:16:27.451562 20001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:16:27.452447 19992 server_base.cc:1034] running on GCE node
I20240503 02:16:27.453155 19992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:16:27.454298 19992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:16:27.455536 19992 hybrid_clock.cc:648] HybridClock initialized: now 1714702587455468 us; error 63 us; skew 500 ppm
I20240503 02:16:27.455933 19992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:16:27.458587 19992 webserver.cc:460] Webserver started at http://127.22.108.8:41063/ using document root <none> and password file <none>
I20240503 02:16:27.459306 19992 fs_manager.cc:359] Metadata directory not provided
I20240503 02:16:27.459414 19992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:16:27.465651 19992 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20240503 02:16:27.469265 20007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:16:27.471021 19992 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20240503 02:16:27.471194 19992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "e9c890f17c8e49cf9f411caeb4df18c6"
format_stamp: "Formatted at 2024-05-03 02:16:08 on dist-test-slave-sc15"
I20240503 02:16:27.472323 19992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:16:27.502506 19992 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:16:27.503558 19992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:16:27.503919 19992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:16:27.505173 19992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:16:27.506820 19992 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:16:27.506901 19992 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:16:27.506994 19992 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:16:27.507036 19992 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:16:27.541610 19992 rpc_server.cc:307] RPC server started. Bound to: 127.22.108.8:32871
I20240503 02:16:27.541684 20119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.108.8:32871 every 8 connection(s)
I20240503 02:16:27.543084 19992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Onrea/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714702278677268-22960-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 02:16:27.550184 22960 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1Onrea/build/asan/bin/kudu as pid 19992
I20240503 02:16:27.561000 20120 heartbeater.cc:347] Connected to a master server at 127.22.108.62:34029
I20240503 02:16:27.561410 20120 heartbeater.cc:464] Registering TS with master...
I20240503 02:16:27.562348 20120 heartbeater.cc:511] Master 127.22.108.62:34029 requested a full tablet report, sending...
I20240503 02:16:27.563906 17616 ts_manager.cc:194] Re-registered known tserver with Master: e9c890f17c8e49cf9f411caeb4df18c6 (127.22.108.8:32871)
I20240503 02:16:27.564746 17616 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.108.8:37621
I20240503 02:16:27.591136 17895 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0fda016a9d3b4f238c0ed4bcf47a00b1"
dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
from {username='slave'} at 127.0.0.1:37608
I20240503 02:16:27.591396 17895 raft_consensus.cc:482] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:27.591517 17895 raft_consensus.cc:3037] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.594127 17895 raft_consensus.cc:504] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:27.595173 17764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fda016a9d3b4f238c0ed4bcf47a00b1" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "836b43a7ee9146948319b8d37434b630"
I20240503 02:16:27.595373 17764 raft_consensus.cc:3037] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.595412 18024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fda016a9d3b4f238c0ed4bcf47a00b1" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:27.595602 18024 raft_consensus.cc:3032] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:16:27.595671 18024 raft_consensus.cc:729] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Running, Role: LEADER
I20240503 02:16:27.595970 18024 consensus_queue.cc:260] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 1cd8517d70d94ae494e713df7eb61b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:27.596462 18024 raft_consensus.cc:3037] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.597476 17764 raft_consensus.cc:2454] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 2.
I20240503 02:16:27.597862 17895 leader_election.cc:288] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 2 election: Requested vote from peers 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837), 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759)
I20240503 02:16:27.597980 17828 leader_election.cc:302] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:27.598219 19547 raft_consensus.cc:2781] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:16:27.598999 18024 raft_consensus.cc:2454] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 1cd8517d70d94ae494e713df7eb61b12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 2.
I20240503 02:16:27.599558 17895 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "736e23a32ac944e2825262be0f7d3992"
dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
from {username='slave'} at 127.0.0.1:37608
I20240503 02:16:27.599786 19547 raft_consensus.cc:686] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Becoming Leader. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:27.599820 17895 raft_consensus.cc:465] T 736e23a32ac944e2825262be0f7d3992 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 02:16:27.600101 19547 consensus_queue.cc:237] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:27.601258 17893 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "353ecf71ac51445087fa3f07e2836db5"
dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
from {username='slave'} at 127.0.0.1:37608
I20240503 02:16:27.601507 17893 raft_consensus.cc:482] T 353ecf71ac51445087fa3f07e2836db5 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:27.601635 17893 raft_consensus.cc:3037] T 353ecf71ac51445087fa3f07e2836db5 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.605808 17893 raft_consensus.cc:504] T 353ecf71ac51445087fa3f07e2836db5 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:27.608587 17764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "353ecf71ac51445087fa3f07e2836db5" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "836b43a7ee9146948319b8d37434b630"
I20240503 02:16:27.608893 17764 raft_consensus.cc:3037] T 353ecf71ac51445087fa3f07e2836db5 P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.611964 17764 raft_consensus.cc:2454] T 353ecf71ac51445087fa3f07e2836db5 P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 2.
I20240503 02:16:27.612643 17828 leader_election.cc:302] T 353ecf71ac51445087fa3f07e2836db5 P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:27.613449 17893 leader_election.cc:288] T 353ecf71ac51445087fa3f07e2836db5 P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 2 election: Requested vote from peers 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837), 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703)
I20240503 02:16:27.613718 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "353ecf71ac51445087fa3f07e2836db5" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431"
I20240503 02:16:27.613936 18155 raft_consensus.cc:3032] T 353ecf71ac51445087fa3f07e2836db5 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:16:27.614071 18155 raft_consensus.cc:729] T 353ecf71ac51445087fa3f07e2836db5 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4325d7d0ed484ad78a6ee54e9a003431, State: Running, Role: LEADER
I20240503 02:16:27.614390 18155 consensus_queue.cc:260] T 353ecf71ac51445087fa3f07e2836db5 P 4325d7d0ed484ad78a6ee54e9a003431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:27.614544 17895 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bb6aa5daccb6436fb7910d2d28244b7c"
dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
from {username='slave'} at 127.0.0.1:37608
I20240503 02:16:27.614732 17895 raft_consensus.cc:465] T bb6aa5daccb6436fb7910d2d28244b7c P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 02:16:27.614989 18155 raft_consensus.cc:3037] T 353ecf71ac51445087fa3f07e2836db5 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.615677 20129 raft_consensus.cc:2781] T 353ecf71ac51445087fa3f07e2836db5 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:16:27.616099 20129 raft_consensus.cc:686] T 353ecf71ac51445087fa3f07e2836db5 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Becoming Leader. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:27.616392 20129 consensus_queue.cc:237] T 353ecf71ac51445087fa3f07e2836db5 P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:27.617234 17894 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3fd3dd4b53b5463a8bc8c166ff355ebe"
dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
from {username='slave'} at 127.0.0.1:37608
I20240503 02:16:27.617431 17894 raft_consensus.cc:482] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:27.617571 17894 raft_consensus.cc:3037] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.622129 18155 raft_consensus.cc:2454] T 353ecf71ac51445087fa3f07e2836db5 P 4325d7d0ed484ad78a6ee54e9a003431 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 2.
I20240503 02:16:27.622735 17894 raft_consensus.cc:504] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:27.623873 17894 leader_election.cc:288] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 2 election: Requested vote from peers 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759), 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837)
I20240503 02:16:27.624063 18024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd3dd4b53b5463a8bc8c166ff355ebe" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:27.624310 18024 raft_consensus.cc:3037] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.624941 19859 heartbeater.cc:503] Master 127.22.108.62:34029 was elected leader, sending a full tablet report...
I20240503 02:16:27.626678 17764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd3dd4b53b5463a8bc8c166ff355ebe" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "836b43a7ee9146948319b8d37434b630"
I20240503 02:16:27.626945 17764 raft_consensus.cc:3032] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 836b43a7ee9146948319b8d37434b630 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:16:27.627022 18024 raft_consensus.cc:2454] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 1cd8517d70d94ae494e713df7eb61b12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 2.
I20240503 02:16:27.627074 17764 raft_consensus.cc:729] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 836b43a7ee9146948319b8d37434b630 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Running, Role: LEADER
I20240503 02:16:27.627409 17764 consensus_queue.cc:260] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 836b43a7ee9146948319b8d37434b630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:27.627923 17831 leader_election.cc:302] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 1cd8517d70d94ae494e713df7eb61b12; no voters:
I20240503 02:16:27.627959 17764 raft_consensus.cc:3037] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.628203 20129 raft_consensus.cc:2781] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:16:27.628429 20129 raft_consensus.cc:686] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Becoming Leader. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:27.628659 20129 consensus_queue.cc:237] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:27.629356 17616 catalog_manager.cc:5484] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 0f3838ee7ad04e94aa65ac750dad0833 reported cstate change: term changed from 1 to 2, leader changed from 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3) to 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2). New cstate: current_term: 2 leader_uuid: "0f3838ee7ad04e94aa65ac750dad0833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } }
I20240503 02:16:27.630131 17894 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ae8c277b101444af870f4f9fd73d915b"
dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
from {username='slave'} at 127.0.0.1:37608
I20240503 02:16:27.630350 17894 raft_consensus.cc:482] T ae8c277b101444af870f4f9fd73d915b P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:27.630489 17894 raft_consensus.cc:3037] T ae8c277b101444af870f4f9fd73d915b P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.638247 17894 raft_consensus.cc:504] T ae8c277b101444af870f4f9fd73d915b P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:27.642488 17764 raft_consensus.cc:2454] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 2.
I20240503 02:16:27.643607 17894 leader_election.cc:288] T ae8c277b101444af870f4f9fd73d915b P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 2 election: Requested vote from peers 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703), 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759)
I20240503 02:16:27.643649 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae8c277b101444af870f4f9fd73d915b" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431"
I20240503 02:16:27.643765 18025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae8c277b101444af870f4f9fd73d915b" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:27.643893 18155 raft_consensus.cc:3032] T ae8c277b101444af870f4f9fd73d915b P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:16:27.644003 18155 raft_consensus.cc:729] T ae8c277b101444af870f4f9fd73d915b P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4325d7d0ed484ad78a6ee54e9a003431, State: Running, Role: LEADER
I20240503 02:16:27.644001 18025 raft_consensus.cc:3037] T ae8c277b101444af870f4f9fd73d915b P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.644296 18155 consensus_queue.cc:260] T ae8c277b101444af870f4f9fd73d915b P 4325d7d0ed484ad78a6ee54e9a003431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:27.644659 18155 raft_consensus.cc:3037] T ae8c277b101444af870f4f9fd73d915b P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.646606 18025 raft_consensus.cc:2454] T ae8c277b101444af870f4f9fd73d915b P 1cd8517d70d94ae494e713df7eb61b12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 2.
I20240503 02:16:27.647030 17831 leader_election.cc:302] T ae8c277b101444af870f4f9fd73d915b P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 1cd8517d70d94ae494e713df7eb61b12; no voters:
I20240503 02:16:27.647377 19547 raft_consensus.cc:2781] T ae8c277b101444af870f4f9fd73d915b P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:16:27.647406 18155 raft_consensus.cc:2454] T ae8c277b101444af870f4f9fd73d915b P 4325d7d0ed484ad78a6ee54e9a003431 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 2.
I20240503 02:16:27.647516 19547 raft_consensus.cc:686] T ae8c277b101444af870f4f9fd73d915b P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Becoming Leader. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:27.647799 19547 consensus_queue.cc:237] T ae8c277b101444af870f4f9fd73d915b P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:27.648870 17894 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "49a29f557c60494c855ba25758b6518b"
dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
from {username='slave'} at 127.0.0.1:37608
I20240503 02:16:27.649062 17894 raft_consensus.cc:465] T 49a29f557c60494c855ba25758b6518b P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 02:16:27.649977 17894 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3d114f397e44d07a729f6eb09f279cc"
dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
from {username='slave'} at 127.0.0.1:37608
I20240503 02:16:27.650168 17894 raft_consensus.cc:482] T d3d114f397e44d07a729f6eb09f279cc P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:27.650300 17894 raft_consensus.cc:3037] T d3d114f397e44d07a729f6eb09f279cc P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.652256 17894 raft_consensus.cc:504] T d3d114f397e44d07a729f6eb09f279cc P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:27.653005 17894 leader_election.cc:288] T d3d114f397e44d07a729f6eb09f279cc P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 2 election: Requested vote from peers 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837), 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703)
I20240503 02:16:27.653607 17764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3d114f397e44d07a729f6eb09f279cc" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "836b43a7ee9146948319b8d37434b630"
I20240503 02:16:27.653692 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3d114f397e44d07a729f6eb09f279cc" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431"
I20240503 02:16:27.653820 17764 raft_consensus.cc:3037] T d3d114f397e44d07a729f6eb09f279cc P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.653892 18155 raft_consensus.cc:3032] T d3d114f397e44d07a729f6eb09f279cc P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:16:27.653993 18155 raft_consensus.cc:729] T d3d114f397e44d07a729f6eb09f279cc P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4325d7d0ed484ad78a6ee54e9a003431, State: Running, Role: LEADER
I20240503 02:16:27.654258 18155 consensus_queue.cc:260] T d3d114f397e44d07a729f6eb09f279cc P 4325d7d0ed484ad78a6ee54e9a003431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:27.654659 18155 raft_consensus.cc:3037] T d3d114f397e44d07a729f6eb09f279cc P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.657430 17764 raft_consensus.cc:2454] T d3d114f397e44d07a729f6eb09f279cc P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 2.
I20240503 02:16:27.658974 18155 raft_consensus.cc:2454] T d3d114f397e44d07a729f6eb09f279cc P 4325d7d0ed484ad78a6ee54e9a003431 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 2.
I20240503 02:16:27.659425 17894 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0f8c3cd667594af4969548ba820e0164"
dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
from {username='slave'} at 127.0.0.1:37608
I20240503 02:16:27.659654 17894 raft_consensus.cc:482] T 0f8c3cd667594af4969548ba820e0164 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:27.659816 17894 raft_consensus.cc:3037] T 0f8c3cd667594af4969548ba820e0164 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.659984 17828 leader_election.cc:302] T d3d114f397e44d07a729f6eb09f279cc P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 4325d7d0ed484ad78a6ee54e9a003431; no voters:
I20240503 02:16:27.660676 19547 raft_consensus.cc:2781] T d3d114f397e44d07a729f6eb09f279cc P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:16:27.660872 19547 raft_consensus.cc:686] T d3d114f397e44d07a729f6eb09f279cc P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Becoming Leader. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:27.661280 19547 consensus_queue.cc:237] T d3d114f397e44d07a729f6eb09f279cc P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:27.662622 17894 raft_consensus.cc:504] T 0f8c3cd667594af4969548ba820e0164 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:27.663277 17894 leader_election.cc:288] T 0f8c3cd667594af4969548ba820e0164 P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 2 election: Requested vote from peers 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703), 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759)
I20240503 02:16:27.663750 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f8c3cd667594af4969548ba820e0164" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431"
I20240503 02:16:27.663950 18155 raft_consensus.cc:3037] T 0f8c3cd667594af4969548ba820e0164 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.664276 18025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f8c3cd667594af4969548ba820e0164" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:27.664485 18025 raft_consensus.cc:3032] T 0f8c3cd667594af4969548ba820e0164 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:16:27.664587 18025 raft_consensus.cc:729] T 0f8c3cd667594af4969548ba820e0164 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Running, Role: LEADER
I20240503 02:16:27.664862 18025 consensus_queue.cc:260] T 0f8c3cd667594af4969548ba820e0164 P 1cd8517d70d94ae494e713df7eb61b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:27.665369 18025 raft_consensus.cc:3037] T 0f8c3cd667594af4969548ba820e0164 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.665601 17894 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bcd9329d71504749b82738119bf7d3ef"
dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
from {username='slave'} at 127.0.0.1:37608
I20240503 02:16:27.665777 17894 raft_consensus.cc:465] T bcd9329d71504749b82738119bf7d3ef P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 02:16:27.666535 18155 raft_consensus.cc:2454] T 0f8c3cd667594af4969548ba820e0164 P 4325d7d0ed484ad78a6ee54e9a003431 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 2.
I20240503 02:16:27.666931 17828 leader_election.cc:302] T 0f8c3cd667594af4969548ba820e0164 P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 4325d7d0ed484ad78a6ee54e9a003431; no voters:
I20240503 02:16:27.667186 20131 raft_consensus.cc:2781] T 0f8c3cd667594af4969548ba820e0164 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:16:27.667331 20131 raft_consensus.cc:686] T 0f8c3cd667594af4969548ba820e0164 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Becoming Leader. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:27.667768 17895 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "022740e08fd644da9d1eb22b3a8a66c0"
dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
from {username='slave'} at 127.0.0.1:37608
I20240503 02:16:27.667634 20131 consensus_queue.cc:237] T 0f8c3cd667594af4969548ba820e0164 P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:27.668001 17895 raft_consensus.cc:482] T 022740e08fd644da9d1eb22b3a8a66c0 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:27.668108 17895 raft_consensus.cc:3037] T 022740e08fd644da9d1eb22b3a8a66c0 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.671452 17895 raft_consensus.cc:504] T 022740e08fd644da9d1eb22b3a8a66c0 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:27.672679 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "022740e08fd644da9d1eb22b3a8a66c0" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431"
I20240503 02:16:27.672928 18155 raft_consensus.cc:3032] T 022740e08fd644da9d1eb22b3a8a66c0 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:16:27.673004 18155 raft_consensus.cc:729] T 022740e08fd644da9d1eb22b3a8a66c0 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4325d7d0ed484ad78a6ee54e9a003431, State: Running, Role: LEADER
I20240503 02:16:27.673290 18155 consensus_queue.cc:260] T 022740e08fd644da9d1eb22b3a8a66c0 P 4325d7d0ed484ad78a6ee54e9a003431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:27.673523 18024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "022740e08fd644da9d1eb22b3a8a66c0" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:27.673712 18024 raft_consensus.cc:3037] T 022740e08fd644da9d1eb22b3a8a66c0 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.673784 18155 raft_consensus.cc:3037] T 022740e08fd644da9d1eb22b3a8a66c0 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.675405 18024 raft_consensus.cc:2454] T 022740e08fd644da9d1eb22b3a8a66c0 P 1cd8517d70d94ae494e713df7eb61b12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 2.
I20240503 02:16:27.675716 17895 leader_election.cc:288] T 022740e08fd644da9d1eb22b3a8a66c0 P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 2 election: Requested vote from peers 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703), 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759)
I20240503 02:16:27.676699 18155 raft_consensus.cc:2454] T 022740e08fd644da9d1eb22b3a8a66c0 P 4325d7d0ed484ad78a6ee54e9a003431 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 2.
I20240503 02:16:27.676767 18025 raft_consensus.cc:2454] T 0f8c3cd667594af4969548ba820e0164 P 1cd8517d70d94ae494e713df7eb61b12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 2.
I20240503 02:16:27.675840 17831 leader_election.cc:302] T 022740e08fd644da9d1eb22b3a8a66c0 P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 1cd8517d70d94ae494e713df7eb61b12; no voters:
I20240503 02:16:27.678925 19547 raft_consensus.cc:2781] T 022740e08fd644da9d1eb22b3a8a66c0 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:16:27.679172 19547 raft_consensus.cc:686] T 022740e08fd644da9d1eb22b3a8a66c0 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Becoming Leader. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:27.679469 19547 consensus_queue.cc:237] T 022740e08fd644da9d1eb22b3a8a66c0 P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:27.679741 17895 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c5812cc63a1f4aa2b8efdcabb1bf5b29"
dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
from {username='slave'} at 127.0.0.1:37608
I20240503 02:16:27.679975 17895 raft_consensus.cc:482] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:27.679834 17616 catalog_manager.cc:5484] T 353ecf71ac51445087fa3f07e2836db5 P 0f3838ee7ad04e94aa65ac750dad0833 reported cstate change: term changed from 1 to 2, leader changed from 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4) to 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2). New cstate: current_term: 2 leader_uuid: "0f3838ee7ad04e94aa65ac750dad0833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } }
I20240503 02:16:27.680367 17895 raft_consensus.cc:3037] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.680368 17616 catalog_manager.cc:5484] T ae8c277b101444af870f4f9fd73d915b P 0f3838ee7ad04e94aa65ac750dad0833 reported cstate change: term changed from 1 to 2, leader changed from 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4) to 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2). New cstate: current_term: 2 leader_uuid: "0f3838ee7ad04e94aa65ac750dad0833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } }
I20240503 02:16:27.680785 17616 catalog_manager.cc:5484] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 0f3838ee7ad04e94aa65ac750dad0833 reported cstate change: term changed from 1 to 2, leader changed from 836b43a7ee9146948319b8d37434b630 (127.22.108.1) to 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2). New cstate: current_term: 2 leader_uuid: "0f3838ee7ad04e94aa65ac750dad0833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } }
I20240503 02:16:27.681233 17616 catalog_manager.cc:5484] T 0f8c3cd667594af4969548ba820e0164 P 0f3838ee7ad04e94aa65ac750dad0833 reported cstate change: term changed from 1 to 2, leader changed from 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3) to 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2). New cstate: current_term: 2 leader_uuid: "0f3838ee7ad04e94aa65ac750dad0833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } }
I20240503 02:16:27.681615 17616 catalog_manager.cc:5484] T d3d114f397e44d07a729f6eb09f279cc P 0f3838ee7ad04e94aa65ac750dad0833 reported cstate change: term changed from 1 to 2, leader changed from 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4) to 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2). New cstate: current_term: 2 leader_uuid: "0f3838ee7ad04e94aa65ac750dad0833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } }
I20240503 02:16:27.684625 17895 raft_consensus.cc:504] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:27.686103 18025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5812cc63a1f4aa2b8efdcabb1bf5b29" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:27.686363 18025 raft_consensus.cc:3032] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:16:27.686493 18025 raft_consensus.cc:729] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Running, Role: LEADER
I20240503 02:16:27.686815 18025 consensus_queue.cc:260] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 1cd8517d70d94ae494e713df7eb61b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:27.687296 18025 raft_consensus.cc:3037] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.689440 18025 raft_consensus.cc:2454] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 1cd8517d70d94ae494e713df7eb61b12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 2.
I20240503 02:16:27.689801 17831 leader_election.cc:302] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 1cd8517d70d94ae494e713df7eb61b12; no voters:
I20240503 02:16:27.690099 19547 raft_consensus.cc:2781] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:16:27.690253 19547 raft_consensus.cc:686] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Becoming Leader. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:27.690493 19547 consensus_queue.cc:237] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:27.690807 17764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5812cc63a1f4aa2b8efdcabb1bf5b29" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "836b43a7ee9146948319b8d37434b630"
I20240503 02:16:27.690658 17616 catalog_manager.cc:5484] T 022740e08fd644da9d1eb22b3a8a66c0 P 0f3838ee7ad04e94aa65ac750dad0833 reported cstate change: term changed from 1 to 2, leader changed from 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4) to 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2). New cstate: current_term: 2 leader_uuid: "0f3838ee7ad04e94aa65ac750dad0833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } }
I20240503 02:16:27.691044 17764 raft_consensus.cc:3037] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.696028 17764 raft_consensus.cc:2454] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 2.
I20240503 02:16:27.696637 17895 leader_election.cc:288] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 2 election: Requested vote from peers 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759), 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837)
I20240503 02:16:27.697867 17895 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0122399369d14e57a55a1415c710e72f"
dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
from {username='slave'} at 127.0.0.1:37608
I20240503 02:16:27.698055 17895 raft_consensus.cc:482] T 0122399369d14e57a55a1415c710e72f P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:27.698168 17895 raft_consensus.cc:3037] T 0122399369d14e57a55a1415c710e72f P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.699474 17616 catalog_manager.cc:5484] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 0f3838ee7ad04e94aa65ac750dad0833 reported cstate change: term changed from 1 to 2, leader changed from 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3) to 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2). New cstate: current_term: 2 leader_uuid: "0f3838ee7ad04e94aa65ac750dad0833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } }
I20240503 02:16:27.700652 17895 raft_consensus.cc:504] T 0122399369d14e57a55a1415c710e72f P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:27.701412 17895 leader_election.cc:288] T 0122399369d14e57a55a1415c710e72f P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 2 election: Requested vote from peers 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703), 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759)
I20240503 02:16:27.701750 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0122399369d14e57a55a1415c710e72f" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431"
I20240503 02:16:27.701987 18155 raft_consensus.cc:3037] T 0122399369d14e57a55a1415c710e72f P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.702361 17894 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7c160a83bd434c56ab99f86d6df68317"
dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
from {username='slave'} at 127.0.0.1:37608
I20240503 02:16:27.702562 17894 raft_consensus.cc:482] T 7c160a83bd434c56ab99f86d6df68317 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:27.702672 17894 raft_consensus.cc:3037] T 7c160a83bd434c56ab99f86d6df68317 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.704583 18155 raft_consensus.cc:2454] T 0122399369d14e57a55a1415c710e72f P 4325d7d0ed484ad78a6ee54e9a003431 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 2.
I20240503 02:16:27.704568 17894 raft_consensus.cc:504] T 7c160a83bd434c56ab99f86d6df68317 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:27.705300 18025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0122399369d14e57a55a1415c710e72f" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:27.705521 18025 raft_consensus.cc:3032] T 0122399369d14e57a55a1415c710e72f P 1cd8517d70d94ae494e713df7eb61b12 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:16:27.705622 18025 raft_consensus.cc:729] T 0122399369d14e57a55a1415c710e72f P 1cd8517d70d94ae494e713df7eb61b12 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Running, Role: LEADER
I20240503 02:16:27.705863 18025 consensus_queue.cc:260] T 0122399369d14e57a55a1415c710e72f P 1cd8517d70d94ae494e713df7eb61b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:27.706127 18024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c160a83bd434c56ab99f86d6df68317" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:27.706231 18025 raft_consensus.cc:3037] T 0122399369d14e57a55a1415c710e72f P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.706347 18024 raft_consensus.cc:3037] T 7c160a83bd434c56ab99f86d6df68317 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.708416 18024 raft_consensus.cc:2454] T 7c160a83bd434c56ab99f86d6df68317 P 1cd8517d70d94ae494e713df7eb61b12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 2.
I20240503 02:16:27.708851 17831 leader_election.cc:302] T 7c160a83bd434c56ab99f86d6df68317 P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 1cd8517d70d94ae494e713df7eb61b12; no voters:
I20240503 02:16:27.709317 17828 leader_election.cc:302] T 0122399369d14e57a55a1415c710e72f P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 4325d7d0ed484ad78a6ee54e9a003431; no voters:
I20240503 02:16:27.709018 18025 raft_consensus.cc:2454] T 0122399369d14e57a55a1415c710e72f P 1cd8517d70d94ae494e713df7eb61b12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 2.
I20240503 02:16:27.709686 17764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c160a83bd434c56ab99f86d6df68317" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "836b43a7ee9146948319b8d37434b630"
I20240503 02:16:27.709880 17764 raft_consensus.cc:3032] T 7c160a83bd434c56ab99f86d6df68317 P 836b43a7ee9146948319b8d37434b630 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:16:27.709988 17764 raft_consensus.cc:729] T 7c160a83bd434c56ab99f86d6df68317 P 836b43a7ee9146948319b8d37434b630 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Running, Role: LEADER
I20240503 02:16:27.710227 17764 consensus_queue.cc:260] T 7c160a83bd434c56ab99f86d6df68317 P 836b43a7ee9146948319b8d37434b630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:27.710484 17764 raft_consensus.cc:3037] T 7c160a83bd434c56ab99f86d6df68317 P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.712190 17764 raft_consensus.cc:2454] T 7c160a83bd434c56ab99f86d6df68317 P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 2.
I20240503 02:16:27.712536 20129 raft_consensus.cc:2781] T 0122399369d14e57a55a1415c710e72f P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:16:27.712677 20129 raft_consensus.cc:686] T 0122399369d14e57a55a1415c710e72f P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Becoming Leader. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:27.712755 17894 leader_election.cc:288] T 7c160a83bd434c56ab99f86d6df68317 P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 2 election: Requested vote from peers 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837), 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759)
I20240503 02:16:27.712970 20129 consensus_queue.cc:237] T 0122399369d14e57a55a1415c710e72f P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:27.713436 19547 raft_consensus.cc:2781] T 7c160a83bd434c56ab99f86d6df68317 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:16:27.713932 17895 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "07ffe3a634c14bd5b0fce34355a5ee4f"
dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
from {username='slave'} at 127.0.0.1:37608
I20240503 02:16:27.714128 17895 raft_consensus.cc:465] T 07ffe3a634c14bd5b0fce34355a5ee4f P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 02:16:27.715370 17894 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6bc5c6531d5144bf80f1b388f23703e9"
dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
from {username='slave'} at 127.0.0.1:37608
I20240503 02:16:27.715552 17894 raft_consensus.cc:482] T 6bc5c6531d5144bf80f1b388f23703e9 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:27.715668 17894 raft_consensus.cc:3037] T 6bc5c6531d5144bf80f1b388f23703e9 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.720419 17894 raft_consensus.cc:504] T 6bc5c6531d5144bf80f1b388f23703e9 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:27.721551 17894 leader_election.cc:288] T 6bc5c6531d5144bf80f1b388f23703e9 P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 2 election: Requested vote from peers 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837), 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759)
I20240503 02:16:27.721545 17764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bc5c6531d5144bf80f1b388f23703e9" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "836b43a7ee9146948319b8d37434b630"
I20240503 02:16:27.721745 17764 raft_consensus.cc:3032] T 6bc5c6531d5144bf80f1b388f23703e9 P 836b43a7ee9146948319b8d37434b630 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:16:27.721861 17764 raft_consensus.cc:729] T 6bc5c6531d5144bf80f1b388f23703e9 P 836b43a7ee9146948319b8d37434b630 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Running, Role: LEADER
I20240503 02:16:27.722048 18025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bc5c6531d5144bf80f1b388f23703e9" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:27.722122 17764 consensus_queue.cc:260] T 6bc5c6531d5144bf80f1b388f23703e9 P 836b43a7ee9146948319b8d37434b630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:27.722250 18025 raft_consensus.cc:3037] T 6bc5c6531d5144bf80f1b388f23703e9 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.722493 17764 raft_consensus.cc:3037] T 6bc5c6531d5144bf80f1b388f23703e9 P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.722719 17895 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6ccff572af764f51938873b46b6622d9"
dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
from {username='slave'} at 127.0.0.1:37608
I20240503 02:16:27.723027 17895 raft_consensus.cc:482] T 6ccff572af764f51938873b46b6622d9 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:27.723196 17895 raft_consensus.cc:3037] T 6ccff572af764f51938873b46b6622d9 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.725251 18025 raft_consensus.cc:2454] T 6bc5c6531d5144bf80f1b388f23703e9 P 1cd8517d70d94ae494e713df7eb61b12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 2.
I20240503 02:16:27.725685 17831 leader_election.cc:302] T 6bc5c6531d5144bf80f1b388f23703e9 P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 1cd8517d70d94ae494e713df7eb61b12; no voters:
I20240503 02:16:27.725991 17764 raft_consensus.cc:2454] T 6bc5c6531d5144bf80f1b388f23703e9 P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 2.
I20240503 02:16:27.726017 20131 raft_consensus.cc:2781] T 6bc5c6531d5144bf80f1b388f23703e9 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:16:27.726181 20131 raft_consensus.cc:686] T 6bc5c6531d5144bf80f1b388f23703e9 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Becoming Leader. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:27.726069 17895 raft_consensus.cc:504] T 6ccff572af764f51938873b46b6622d9 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:27.726467 20131 consensus_queue.cc:237] T 6bc5c6531d5144bf80f1b388f23703e9 P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:27.726830 17894 raft_consensus.cc:1226] T 6ccff572af764f51938873b46b6622d9 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Rejecting Update request from peer 836b43a7ee9146948319b8d37434b630 for earlier term 1. Current term is 2. Ops: []
I20240503 02:16:27.727133 19547 raft_consensus.cc:686] T 7c160a83bd434c56ab99f86d6df68317 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Becoming Leader. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:27.727386 19547 consensus_queue.cc:237] T 7c160a83bd434c56ab99f86d6df68317 P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:27.727504 19596 consensus_queue.cc:1046] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:16:27.728221 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ccff572af764f51938873b46b6622d9" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431"
I20240503 02:16:27.728410 18155 raft_consensus.cc:3037] T 6ccff572af764f51938873b46b6622d9 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.730991 18155 raft_consensus.cc:2454] T 6ccff572af764f51938873b46b6622d9 P 4325d7d0ed484ad78a6ee54e9a003431 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 2.
I20240503 02:16:27.737452 17828 leader_election.cc:302] T 6ccff572af764f51938873b46b6622d9 P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 4325d7d0ed484ad78a6ee54e9a003431; no voters:
I20240503 02:16:27.738579 20131 raft_consensus.cc:2781] T 6ccff572af764f51938873b46b6622d9 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:16:27.738946 20131 raft_consensus.cc:686] T 6ccff572af764f51938873b46b6622d9 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Becoming Leader. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:27.739334 20131 consensus_queue.cc:237] T 6ccff572af764f51938873b46b6622d9 P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:27.740325 17764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ccff572af764f51938873b46b6622d9" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "836b43a7ee9146948319b8d37434b630"
I20240503 02:16:27.740950 17895 leader_election.cc:288] T 6ccff572af764f51938873b46b6622d9 P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 2 election: Requested vote from peers 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703), 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837)
I20240503 02:16:27.742209 17894 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9ac1c650360f4088ba5cc339cee6d45e"
dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
from {username='slave'} at 127.0.0.1:37608
I20240503 02:16:27.742445 17894 raft_consensus.cc:482] T 9ac1c650360f4088ba5cc339cee6d45e P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:27.742596 17894 raft_consensus.cc:3037] T 9ac1c650360f4088ba5cc339cee6d45e P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.744971 17894 raft_consensus.cc:504] T 9ac1c650360f4088ba5cc339cee6d45e P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:27.745697 17894 leader_election.cc:288] T 9ac1c650360f4088ba5cc339cee6d45e P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 2 election: Requested vote from peers 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759), 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837)
I20240503 02:16:27.745790 17764 raft_consensus.cc:3032] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:16:27.745913 17764 raft_consensus.cc:729] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Running, Role: LEADER
I20240503 02:16:27.746121 18025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac1c650360f4088ba5cc339cee6d45e" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:27.746176 17764 consensus_queue.cc:260] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:27.746347 18025 raft_consensus.cc:3032] T 9ac1c650360f4088ba5cc339cee6d45e P 1cd8517d70d94ae494e713df7eb61b12 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:16:27.746438 18025 raft_consensus.cc:729] T 9ac1c650360f4088ba5cc339cee6d45e P 1cd8517d70d94ae494e713df7eb61b12 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Running, Role: LEADER
I20240503 02:16:27.746380 17762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac1c650360f4088ba5cc339cee6d45e" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "836b43a7ee9146948319b8d37434b630"
I20240503 02:16:27.746521 17764 raft_consensus.cc:3037] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.746582 17762 raft_consensus.cc:3037] T 9ac1c650360f4088ba5cc339cee6d45e P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.746731 18025 consensus_queue.cc:260] T 9ac1c650360f4088ba5cc339cee6d45e P 1cd8517d70d94ae494e713df7eb61b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:27.747104 18025 raft_consensus.cc:3037] T 9ac1c650360f4088ba5cc339cee6d45e P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.749577 17762 raft_consensus.cc:2454] T 9ac1c650360f4088ba5cc339cee6d45e P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 2.
I20240503 02:16:27.749853 18025 raft_consensus.cc:2454] T 9ac1c650360f4088ba5cc339cee6d45e P 1cd8517d70d94ae494e713df7eb61b12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 2.
I20240503 02:16:27.750243 17831 leader_election.cc:302] T 9ac1c650360f4088ba5cc339cee6d45e P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 1cd8517d70d94ae494e713df7eb61b12; no voters:
I20240503 02:16:27.750667 19547 raft_consensus.cc:2781] T 9ac1c650360f4088ba5cc339cee6d45e P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:16:27.750922 19547 raft_consensus.cc:686] T 9ac1c650360f4088ba5cc339cee6d45e P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Becoming Leader. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:27.751155 17894 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a8cc6cfa2b8240c9981a692e0896ef50"
dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
from {username='slave'} at 127.0.0.1:37608
I20240503 02:16:27.751195 19547 consensus_queue.cc:237] T 9ac1c650360f4088ba5cc339cee6d45e P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:27.751349 17894 raft_consensus.cc:465] T a8cc6cfa2b8240c9981a692e0896ef50 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 02:16:27.752247 17894 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2625f9a7afe74874bd6ccb05d6d6e673"
dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
from {username='slave'} at 127.0.0.1:37608
I20240503 02:16:27.752472 17894 raft_consensus.cc:482] T 2625f9a7afe74874bd6ccb05d6d6e673 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:27.752593 17894 raft_consensus.cc:3037] T 2625f9a7afe74874bd6ccb05d6d6e673 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.752753 17616 catalog_manager.cc:5484] T 0122399369d14e57a55a1415c710e72f P 0f3838ee7ad04e94aa65ac750dad0833 reported cstate change: term changed from 1 to 2, leader changed from 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3) to 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2). New cstate: current_term: 2 leader_uuid: "0f3838ee7ad04e94aa65ac750dad0833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } }
I20240503 02:16:27.753321 17616 catalog_manager.cc:5484] T 7c160a83bd434c56ab99f86d6df68317 P 0f3838ee7ad04e94aa65ac750dad0833 reported cstate change: term changed from 1 to 2, leader changed from 836b43a7ee9146948319b8d37434b630 (127.22.108.1) to 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2). New cstate: current_term: 2 leader_uuid: "0f3838ee7ad04e94aa65ac750dad0833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } }
I20240503 02:16:27.756260 17764 raft_consensus.cc:2454] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 2.
I20240503 02:16:27.756736 17894 raft_consensus.cc:504] T 2625f9a7afe74874bd6ccb05d6d6e673 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
W20240503 02:16:27.758251 20147 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20240503 02:16:27.758185 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2625f9a7afe74874bd6ccb05d6d6e673" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431"
I20240503 02:16:27.758466 18155 raft_consensus.cc:3032] T 2625f9a7afe74874bd6ccb05d6d6e673 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:16:27.758560 18155 raft_consensus.cc:729] T 2625f9a7afe74874bd6ccb05d6d6e673 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4325d7d0ed484ad78a6ee54e9a003431, State: Running, Role: LEADER
I20240503 02:16:27.758976 18155 consensus_queue.cc:260] T 2625f9a7afe74874bd6ccb05d6d6e673 P 4325d7d0ed484ad78a6ee54e9a003431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:27.759442 18155 raft_consensus.cc:3037] T 2625f9a7afe74874bd6ccb05d6d6e673 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.763710 18025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2625f9a7afe74874bd6ccb05d6d6e673" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:27.764075 18025 raft_consensus.cc:3037] T 2625f9a7afe74874bd6ccb05d6d6e673 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.767810 18025 raft_consensus.cc:2454] T 2625f9a7afe74874bd6ccb05d6d6e673 P 1cd8517d70d94ae494e713df7eb61b12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 2.
I20240503 02:16:27.768399 17831 leader_election.cc:302] T 2625f9a7afe74874bd6ccb05d6d6e673 P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 1cd8517d70d94ae494e713df7eb61b12; no voters:
I20240503 02:16:27.768623 17894 leader_election.cc:288] T 2625f9a7afe74874bd6ccb05d6d6e673 P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 2 election: Requested vote from peers 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703), 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759)
I20240503 02:16:27.769001 20131 raft_consensus.cc:2781] T 2625f9a7afe74874bd6ccb05d6d6e673 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:16:27.769234 20131 raft_consensus.cc:686] T 2625f9a7afe74874bd6ccb05d6d6e673 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Becoming Leader. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:27.769434 18155 raft_consensus.cc:2454] T 2625f9a7afe74874bd6ccb05d6d6e673 P 4325d7d0ed484ad78a6ee54e9a003431 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 2.
I20240503 02:16:27.769574 20131 consensus_queue.cc:237] T 2625f9a7afe74874bd6ccb05d6d6e673 P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:27.772296 17894 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6315543b56a3474da4a13026ad498f84"
dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
from {username='slave'} at 127.0.0.1:37608
I20240503 02:16:27.772455 17894 raft_consensus.cc:482] T 6315543b56a3474da4a13026ad498f84 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:27.772573 17894 raft_consensus.cc:3037] T 6315543b56a3474da4a13026ad498f84 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.773114 17616 catalog_manager.cc:5484] T 9ac1c650360f4088ba5cc339cee6d45e P 0f3838ee7ad04e94aa65ac750dad0833 reported cstate change: term changed from 1 to 2, leader changed from 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3) to 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2). New cstate: current_term: 2 leader_uuid: "0f3838ee7ad04e94aa65ac750dad0833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } }
I20240503 02:16:27.773619 17616 catalog_manager.cc:5484] T 6ccff572af764f51938873b46b6622d9 P 0f3838ee7ad04e94aa65ac750dad0833 reported cstate change: term changed from 1 to 2, leader changed from 836b43a7ee9146948319b8d37434b630 (127.22.108.1) to 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2). New cstate: current_term: 2 leader_uuid: "0f3838ee7ad04e94aa65ac750dad0833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } }
I20240503 02:16:27.774027 17616 catalog_manager.cc:5484] T 6bc5c6531d5144bf80f1b388f23703e9 P 0f3838ee7ad04e94aa65ac750dad0833 reported cstate change: term changed from 1 to 2, leader changed from 836b43a7ee9146948319b8d37434b630 (127.22.108.1) to 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2). New cstate: current_term: 2 leader_uuid: "0f3838ee7ad04e94aa65ac750dad0833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } }
I20240503 02:16:27.774783 17894 raft_consensus.cc:504] T 6315543b56a3474da4a13026ad498f84 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:27.775595 17894 leader_election.cc:288] T 6315543b56a3474da4a13026ad498f84 P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 2 election: Requested vote from peers 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837), 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759)
I20240503 02:16:27.775828 17764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6315543b56a3474da4a13026ad498f84" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "836b43a7ee9146948319b8d37434b630"
I20240503 02:16:27.776027 17764 raft_consensus.cc:3032] T 6315543b56a3474da4a13026ad498f84 P 836b43a7ee9146948319b8d37434b630 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:16:27.776113 17764 raft_consensus.cc:729] T 6315543b56a3474da4a13026ad498f84 P 836b43a7ee9146948319b8d37434b630 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Running, Role: LEADER
I20240503 02:16:27.776394 18025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6315543b56a3474da4a13026ad498f84" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:27.776347 17764 consensus_queue.cc:260] T 6315543b56a3474da4a13026ad498f84 P 836b43a7ee9146948319b8d37434b630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:27.776571 18025 raft_consensus.cc:3037] T 6315543b56a3474da4a13026ad498f84 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.776798 17764 raft_consensus.cc:3037] T 6315543b56a3474da4a13026ad498f84 P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.779237 18025 raft_consensus.cc:2454] T 6315543b56a3474da4a13026ad498f84 P 1cd8517d70d94ae494e713df7eb61b12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 2.
I20240503 02:16:27.779599 17831 leader_election.cc:302] T 6315543b56a3474da4a13026ad498f84 P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 1cd8517d70d94ae494e713df7eb61b12; no voters:
I20240503 02:16:27.780104 20144 raft_consensus.cc:2781] T 6315543b56a3474da4a13026ad498f84 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:16:27.780206 17764 raft_consensus.cc:2454] T 6315543b56a3474da4a13026ad498f84 P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 2.
I20240503 02:16:27.780289 20144 raft_consensus.cc:686] T 6315543b56a3474da4a13026ad498f84 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Becoming Leader. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:27.780609 20144 consensus_queue.cc:237] T 6315543b56a3474da4a13026ad498f84 P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:27.781335 17894 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c0300c0a94654f08b08f0ef0cea1d6be"
dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
from {username='slave'} at 127.0.0.1:37608
I20240503 02:16:27.781554 17894 raft_consensus.cc:482] T c0300c0a94654f08b08f0ef0cea1d6be P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:27.781678 17894 raft_consensus.cc:3037] T c0300c0a94654f08b08f0ef0cea1d6be P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.784345 17616 catalog_manager.cc:5484] T 2625f9a7afe74874bd6ccb05d6d6e673 P 0f3838ee7ad04e94aa65ac750dad0833 reported cstate change: term changed from 1 to 2, leader changed from 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4) to 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2). New cstate: current_term: 2 leader_uuid: "0f3838ee7ad04e94aa65ac750dad0833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } }
I20240503 02:16:27.784749 17616 catalog_manager.cc:5484] T 6315543b56a3474da4a13026ad498f84 P 0f3838ee7ad04e94aa65ac750dad0833 reported cstate change: term changed from 1 to 2, leader changed from 836b43a7ee9146948319b8d37434b630 (127.22.108.1) to 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2). New cstate: current_term: 2 leader_uuid: "0f3838ee7ad04e94aa65ac750dad0833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } }
I20240503 02:16:27.786065 17894 raft_consensus.cc:504] T c0300c0a94654f08b08f0ef0cea1d6be P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:27.786832 17894 leader_election.cc:288] T c0300c0a94654f08b08f0ef0cea1d6be P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 2 election: Requested vote from peers 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759), 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703)
I20240503 02:16:27.788518 18025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0300c0a94654f08b08f0ef0cea1d6be" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:27.788785 18025 raft_consensus.cc:3037] T c0300c0a94654f08b08f0ef0cea1d6be P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.789013 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0300c0a94654f08b08f0ef0cea1d6be" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431"
I20240503 02:16:27.789328 18155 raft_consensus.cc:3032] T c0300c0a94654f08b08f0ef0cea1d6be P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:16:27.789535 18155 raft_consensus.cc:729] T c0300c0a94654f08b08f0ef0cea1d6be P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4325d7d0ed484ad78a6ee54e9a003431, State: Running, Role: LEADER
I20240503 02:16:27.789891 17894 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dacfa7037bae46fb9b3d34ba4b8e54b4"
dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
from {username='slave'} at 127.0.0.1:37608
I20240503 02:16:27.790081 17894 raft_consensus.cc:482] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:27.790203 17894 raft_consensus.cc:3037] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.789949 18155 consensus_queue.cc:260] T c0300c0a94654f08b08f0ef0cea1d6be P 4325d7d0ed484ad78a6ee54e9a003431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:27.790591 18155 raft_consensus.cc:3037] T c0300c0a94654f08b08f0ef0cea1d6be P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.791834 18025 raft_consensus.cc:2454] T c0300c0a94654f08b08f0ef0cea1d6be P 1cd8517d70d94ae494e713df7eb61b12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 2.
I20240503 02:16:27.792331 17831 leader_election.cc:302] T c0300c0a94654f08b08f0ef0cea1d6be P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 1cd8517d70d94ae494e713df7eb61b12; no voters:
I20240503 02:16:27.792604 20144 raft_consensus.cc:2781] T c0300c0a94654f08b08f0ef0cea1d6be P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:16:27.792757 20144 raft_consensus.cc:686] T c0300c0a94654f08b08f0ef0cea1d6be P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Becoming Leader. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:27.793083 20144 consensus_queue.cc:237] T c0300c0a94654f08b08f0ef0cea1d6be P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:27.793164 17894 raft_consensus.cc:504] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:27.794181 18155 raft_consensus.cc:2454] T c0300c0a94654f08b08f0ef0cea1d6be P 4325d7d0ed484ad78a6ee54e9a003431 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 2.
I20240503 02:16:27.794260 17764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacfa7037bae46fb9b3d34ba4b8e54b4" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "836b43a7ee9146948319b8d37434b630"
I20240503 02:16:27.794457 17764 raft_consensus.cc:3037] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.796512 17764 raft_consensus.cc:2454] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 2.
I20240503 02:16:27.796927 17828 leader_election.cc:302] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:27.797215 20131 raft_consensus.cc:2781] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:16:27.797350 20131 raft_consensus.cc:686] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Becoming Leader. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:27.797590 20131 consensus_queue.cc:237] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:27.799365 17894 leader_election.cc:288] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 0f3838ee7ad04e94aa65ac750dad0833 [CANDIDATE]: Term 2 election: Requested vote from peers 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837), 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703)
I20240503 02:16:27.799391 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacfa7037bae46fb9b3d34ba4b8e54b4" candidate_uuid: "0f3838ee7ad04e94aa65ac750dad0833" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431"
I20240503 02:16:27.799597 18155 raft_consensus.cc:3032] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:16:27.799683 18155 raft_consensus.cc:729] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4325d7d0ed484ad78a6ee54e9a003431, State: Running, Role: LEADER
I20240503 02:16:27.799964 18155 consensus_queue.cc:260] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 4325d7d0ed484ad78a6ee54e9a003431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:27.800232 18155 raft_consensus.cc:3037] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.804446 18155 raft_consensus.cc:2454] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 4325d7d0ed484ad78a6ee54e9a003431 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3838ee7ad04e94aa65ac750dad0833 in term 2.
I20240503 02:16:27.804963 17616 catalog_manager.cc:5484] T c0300c0a94654f08b08f0ef0cea1d6be P 0f3838ee7ad04e94aa65ac750dad0833 reported cstate change: term changed from 1 to 2, leader changed from 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4) to 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2). New cstate: current_term: 2 leader_uuid: "0f3838ee7ad04e94aa65ac750dad0833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } }
I20240503 02:16:27.811964 17616 catalog_manager.cc:5484] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 0f3838ee7ad04e94aa65ac750dad0833 reported cstate change: term changed from 1 to 2, leader changed from 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4) to 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2). New cstate: current_term: 2 leader_uuid: "0f3838ee7ad04e94aa65ac750dad0833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } }
I20240503 02:16:27.840262 17764 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0fda016a9d3b4f238c0ed4bcf47a00b1"
dest_uuid: "836b43a7ee9146948319b8d37434b630"
from {username='slave'} at 127.0.0.1:50482
I20240503 02:16:27.840520 17764 raft_consensus.cc:482] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:27.840628 17764 raft_consensus.cc:3037] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:16:27.842664 17764 raft_consensus.cc:504] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 836b43a7ee9146948319b8d37434b630 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:27.843333 17764 leader_election.cc:288] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 3 election: Requested vote from peers 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647), 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759)
I20240503 02:16:27.843890 18025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fda016a9d3b4f238c0ed4bcf47a00b1" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:27.844168 18025 raft_consensus.cc:3037] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 1cd8517d70d94ae494e713df7eb61b12 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:16:27.844321 17894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fda016a9d3b4f238c0ed4bcf47a00b1" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
I20240503 02:16:27.844606 17894 raft_consensus.cc:3032] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:16:27.844666 17764 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "954f9a466d8648958df7baa916ea8558"
dest_uuid: "836b43a7ee9146948319b8d37434b630"
from {username='slave'} at 127.0.0.1:50482
I20240503 02:16:27.844738 17894 raft_consensus.cc:729] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:27.844866 17764 raft_consensus.cc:482] T 954f9a466d8648958df7baa916ea8558 P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:27.844992 17764 raft_consensus.cc:3037] T 954f9a466d8648958df7baa916ea8558 P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.845337 17894 consensus_queue.cc:260] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:27.845865 17894 raft_consensus.cc:3037] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:16:27.846971 18025 raft_consensus.cc:2454] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 1cd8517d70d94ae494e713df7eb61b12 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 3.
I20240503 02:16:27.847448 17700 leader_election.cc:302] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cd8517d70d94ae494e713df7eb61b12, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:27.847774 19596 raft_consensus.cc:2781] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 836b43a7ee9146948319b8d37434b630 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:16:27.847949 19596 raft_consensus.cc:686] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 836b43a7ee9146948319b8d37434b630 [term 3 LEADER]: Becoming Leader. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Running, Role: LEADER
I20240503 02:16:27.847841 17764 raft_consensus.cc:504] T 954f9a466d8648958df7baa916ea8558 P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:27.848232 19596 consensus_queue.cc:237] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 836b43a7ee9146948319b8d37434b630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:27.848729 17764 leader_election.cc:288] T 954f9a466d8648958df7baa916ea8558 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 2 election: Requested vote from peers 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759), 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703)
I20240503 02:16:27.849201 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "954f9a466d8648958df7baa916ea8558" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431"
I20240503 02:16:27.849431 18155 raft_consensus.cc:3032] T 954f9a466d8648958df7baa916ea8558 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:16:27.849541 18155 raft_consensus.cc:729] T 954f9a466d8648958df7baa916ea8558 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4325d7d0ed484ad78a6ee54e9a003431, State: Running, Role: LEADER
I20240503 02:16:27.849898 17762 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "353ecf71ac51445087fa3f07e2836db5"
dest_uuid: "836b43a7ee9146948319b8d37434b630"
from {username='slave'} at 127.0.0.1:50482
I20240503 02:16:27.849879 18155 consensus_queue.cc:260] T 954f9a466d8648958df7baa916ea8558 P 4325d7d0ed484ad78a6ee54e9a003431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:27.850134 17762 raft_consensus.cc:482] T 353ecf71ac51445087fa3f07e2836db5 P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:27.850260 17762 raft_consensus.cc:3037] T 353ecf71ac51445087fa3f07e2836db5 P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:16:27.850327 18155 raft_consensus.cc:3037] T 954f9a466d8648958df7baa916ea8558 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.853250 17762 raft_consensus.cc:504] T 353ecf71ac51445087fa3f07e2836db5 P 836b43a7ee9146948319b8d37434b630 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:27.853940 18024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "954f9a466d8648958df7baa916ea8558" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:27.854127 18024 raft_consensus.cc:3037] T 954f9a466d8648958df7baa916ea8558 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.854586 17762 leader_election.cc:288] T 353ecf71ac51445087fa3f07e2836db5 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 3 election: Requested vote from peers 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703), 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647)
I20240503 02:16:27.855527 17763 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bb6aa5daccb6436fb7910d2d28244b7c"
dest_uuid: "836b43a7ee9146948319b8d37434b630"
from {username='slave'} at 127.0.0.1:50482
I20240503 02:16:27.855698 17763 raft_consensus.cc:482] T bb6aa5daccb6436fb7910d2d28244b7c P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:27.855803 17763 raft_consensus.cc:3037] T bb6aa5daccb6436fb7910d2d28244b7c P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.856896 18024 raft_consensus.cc:2454] T 954f9a466d8648958df7baa916ea8558 P 1cd8517d70d94ae494e713df7eb61b12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 2.
I20240503 02:16:27.857728 17895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "353ecf71ac51445087fa3f07e2836db5" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
I20240503 02:16:27.857950 17895 raft_consensus.cc:3032] T 353ecf71ac51445087fa3f07e2836db5 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:16:27.858023 17895 raft_consensus.cc:729] T 353ecf71ac51445087fa3f07e2836db5 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:27.858282 17895 consensus_queue.cc:260] T 353ecf71ac51445087fa3f07e2836db5 P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:27.858570 17895 raft_consensus.cc:3037] T 353ecf71ac51445087fa3f07e2836db5 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:16:27.858675 17763 raft_consensus.cc:504] T bb6aa5daccb6436fb7910d2d28244b7c P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:27.861083 17763 leader_election.cc:288] T bb6aa5daccb6436fb7910d2d28244b7c P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 2 election: Requested vote from peers 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647), 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759)
I20240503 02:16:27.861130 18023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb6aa5daccb6436fb7910d2d28244b7c" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:27.861331 18023 raft_consensus.cc:3037] T bb6aa5daccb6436fb7910d2d28244b7c P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.862211 17761 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3fd3dd4b53b5463a8bc8c166ff355ebe"
dest_uuid: "836b43a7ee9146948319b8d37434b630"
from {username='slave'} at 127.0.0.1:50482
I20240503 02:16:27.862376 17761 raft_consensus.cc:482] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:27.862476 17761 raft_consensus.cc:3037] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:16:27.864429 18023 raft_consensus.cc:2454] T bb6aa5daccb6436fb7910d2d28244b7c P 1cd8517d70d94ae494e713df7eb61b12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 2.
I20240503 02:16:27.865971 17700 leader_election.cc:302] T bb6aa5daccb6436fb7910d2d28244b7c P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cd8517d70d94ae494e713df7eb61b12, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:27.864962 17893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb6aa5daccb6436fb7910d2d28244b7c" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
I20240503 02:16:27.866258 20147 raft_consensus.cc:2781] T bb6aa5daccb6436fb7910d2d28244b7c P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:16:27.866290 17893 raft_consensus.cc:3032] T bb6aa5daccb6436fb7910d2d28244b7c P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:16:27.866413 17893 raft_consensus.cc:729] T bb6aa5daccb6436fb7910d2d28244b7c P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:27.865456 17761 raft_consensus.cc:504] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 836b43a7ee9146948319b8d37434b630 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:27.866720 17893 consensus_queue.cc:260] T bb6aa5daccb6436fb7910d2d28244b7c P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:27.867223 17893 raft_consensus.cc:3037] T bb6aa5daccb6436fb7910d2d28244b7c P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.867722 17892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd3dd4b53b5463a8bc8c166ff355ebe" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
I20240503 02:16:27.867976 17892 raft_consensus.cc:3032] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:16:27.868091 17892 raft_consensus.cc:729] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:27.870291 17895 raft_consensus.cc:2396] T 353ecf71ac51445087fa3f07e2836db5 P 0f3838ee7ad04e94aa65ac750dad0833 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 836b43a7ee9146948319b8d37434b630 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:16:27.872159 17893 raft_consensus.cc:2454] T bb6aa5daccb6436fb7910d2d28244b7c P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 2.
I20240503 02:16:27.872587 18025 raft_consensus.cc:1226] T 954f9a466d8648958df7baa916ea8558 P 1cd8517d70d94ae494e713df7eb61b12 [term 2 FOLLOWER]: Rejecting Update request from peer 4325d7d0ed484ad78a6ee54e9a003431 for earlier term 1. Current term is 2. Ops: []
I20240503 02:16:27.873368 18154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "353ecf71ac51445087fa3f07e2836db5" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431"
I20240503 02:16:27.873577 18154 raft_consensus.cc:3037] T 353ecf71ac51445087fa3f07e2836db5 P 4325d7d0ed484ad78a6ee54e9a003431 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:16:27.876319 18154 raft_consensus.cc:2454] T 353ecf71ac51445087fa3f07e2836db5 P 4325d7d0ed484ad78a6ee54e9a003431 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 3.
I20240503 02:16:27.876915 17697 leader_election.cc:302] T 353ecf71ac51445087fa3f07e2836db5 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4325d7d0ed484ad78a6ee54e9a003431, 836b43a7ee9146948319b8d37434b630; no voters: 0f3838ee7ad04e94aa65ac750dad0833
I20240503 02:16:27.877303 17700 leader_election.cc:302] T 954f9a466d8648958df7baa916ea8558 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cd8517d70d94ae494e713df7eb61b12, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:27.877565 18155 raft_consensus.cc:2454] T 954f9a466d8648958df7baa916ea8558 P 4325d7d0ed484ad78a6ee54e9a003431 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 2.
I20240503 02:16:27.878054 20160 raft_consensus.cc:2781] T 353ecf71ac51445087fa3f07e2836db5 P 836b43a7ee9146948319b8d37434b630 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:16:27.876727 17892 consensus_queue.cc:260] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:27.878291 20160 raft_consensus.cc:686] T 353ecf71ac51445087fa3f07e2836db5 P 836b43a7ee9146948319b8d37434b630 [term 3 LEADER]: Becoming Leader. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Running, Role: LEADER
I20240503 02:16:27.878466 17892 raft_consensus.cc:3037] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:16:27.878650 20160 consensus_queue.cc:237] T 353ecf71ac51445087fa3f07e2836db5 P 836b43a7ee9146948319b8d37434b630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:27.881340 17892 raft_consensus.cc:2396] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 0f3838ee7ad04e94aa65ac750dad0833 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 836b43a7ee9146948319b8d37434b630 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:16:27.881755 17761 leader_election.cc:288] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 3 election: Requested vote from peers 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759), 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647)
I20240503 02:16:27.882850 17760 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "628548a443e64c6fae75070970bba761"
dest_uuid: "836b43a7ee9146948319b8d37434b630"
from {username='slave'} at 127.0.0.1:50482
I20240503 02:16:27.883124 17760 raft_consensus.cc:482] T 628548a443e64c6fae75070970bba761 P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:27.883121 18025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd3dd4b53b5463a8bc8c166ff355ebe" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:27.883265 17760 raft_consensus.cc:3037] T 628548a443e64c6fae75070970bba761 P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.883320 18025 raft_consensus.cc:3037] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 1cd8517d70d94ae494e713df7eb61b12 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:16:27.886469 18025 raft_consensus.cc:2454] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 1cd8517d70d94ae494e713df7eb61b12 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 3.
I20240503 02:16:27.886974 17700 leader_election.cc:302] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cd8517d70d94ae494e713df7eb61b12, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:27.886973 17760 raft_consensus.cc:504] T 628548a443e64c6fae75070970bba761 P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:27.888168 17760 leader_election.cc:288] T 628548a443e64c6fae75070970bba761 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 2 election: Requested vote from peers 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759), 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703)
I20240503 02:16:27.891007 17763 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a8cc6cfa2b8240c9981a692e0896ef50"
dest_uuid: "836b43a7ee9146948319b8d37434b630"
from {username='slave'} at 127.0.0.1:50482
I20240503 02:16:27.891235 17763 raft_consensus.cc:482] T a8cc6cfa2b8240c9981a692e0896ef50 P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:27.891367 17763 raft_consensus.cc:3037] T a8cc6cfa2b8240c9981a692e0896ef50 P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.894273 17894 raft_consensus.cc:2396] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 0f3838ee7ad04e94aa65ac750dad0833 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 836b43a7ee9146948319b8d37434b630 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:16:27.910799 17763 raft_consensus.cc:504] T a8cc6cfa2b8240c9981a692e0896ef50 P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:27.911638 17763 leader_election.cc:288] T a8cc6cfa2b8240c9981a692e0896ef50 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 2 election: Requested vote from peers 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703), 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647)
I20240503 02:16:27.911798 20160 raft_consensus.cc:2781] T 954f9a466d8648958df7baa916ea8558 P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:16:27.912276 20160 raft_consensus.cc:686] T 954f9a466d8648958df7baa916ea8558 P 836b43a7ee9146948319b8d37434b630 [term 2 LEADER]: Becoming Leader. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Running, Role: LEADER
I20240503 02:16:27.912623 20160 consensus_queue.cc:237] T 954f9a466d8648958df7baa916ea8558 P 836b43a7ee9146948319b8d37434b630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:27.915328 17763 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9ac1c650360f4088ba5cc339cee6d45e"
dest_uuid: "836b43a7ee9146948319b8d37434b630"
from {username='slave'} at 127.0.0.1:50482
I20240503 02:16:27.915658 17763 raft_consensus.cc:482] T 9ac1c650360f4088ba5cc339cee6d45e P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:27.915871 17763 raft_consensus.cc:3037] T 9ac1c650360f4088ba5cc339cee6d45e P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:16:27.916285 19596 raft_consensus.cc:2781] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 836b43a7ee9146948319b8d37434b630 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:16:27.916436 19596 raft_consensus.cc:686] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 836b43a7ee9146948319b8d37434b630 [term 3 LEADER]: Becoming Leader. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Running, Role: LEADER
I20240503 02:16:27.916747 19596 consensus_queue.cc:237] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 836b43a7ee9146948319b8d37434b630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:27.921262 17616 catalog_manager.cc:5484] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 836b43a7ee9146948319b8d37434b630 reported cstate change: term changed from 2 to 3, leader changed from 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2) to 836b43a7ee9146948319b8d37434b630 (127.22.108.1). New cstate: current_term: 3 leader_uuid: "836b43a7ee9146948319b8d37434b630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } }
I20240503 02:16:27.922052 17763 raft_consensus.cc:504] T 9ac1c650360f4088ba5cc339cee6d45e P 836b43a7ee9146948319b8d37434b630 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:27.922806 17763 leader_election.cc:288] T 9ac1c650360f4088ba5cc339cee6d45e P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 3 election: Requested vote from peers 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647), 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759)
I20240503 02:16:27.923835 20147 raft_consensus.cc:686] T bb6aa5daccb6436fb7910d2d28244b7c P 836b43a7ee9146948319b8d37434b630 [term 2 LEADER]: Becoming Leader. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Running, Role: LEADER
I20240503 02:16:27.924197 20147 consensus_queue.cc:237] T bb6aa5daccb6436fb7910d2d28244b7c P 836b43a7ee9146948319b8d37434b630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:27.925941 18025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "628548a443e64c6fae75070970bba761" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:27.926175 18025 raft_consensus.cc:3032] T 628548a443e64c6fae75070970bba761 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:16:27.926256 18025 raft_consensus.cc:729] T 628548a443e64c6fae75070970bba761 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Running, Role: LEADER
I20240503 02:16:27.926524 17763 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e53bab9a159147d79a1f96269f1c1201"
dest_uuid: "836b43a7ee9146948319b8d37434b630"
from {username='slave'} at 127.0.0.1:50482
I20240503 02:16:27.926532 18025 consensus_queue.cc:260] T 628548a443e64c6fae75070970bba761 P 1cd8517d70d94ae494e713df7eb61b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:27.926797 17763 raft_consensus.cc:465] T e53bab9a159147d79a1f96269f1c1201 P 836b43a7ee9146948319b8d37434b630 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 02:16:27.926945 18025 raft_consensus.cc:3037] T 628548a443e64c6fae75070970bba761 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:16:27.927906 17697 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.108.4:39703, user_credentials={real_user=slave}} blocked reactor thread for 50997us
I20240503 02:16:27.928658 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "628548a443e64c6fae75070970bba761" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431"
I20240503 02:16:27.928895 18155 raft_consensus.cc:3037] T 628548a443e64c6fae75070970bba761 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.929189 17764 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6ccff572af764f51938873b46b6622d9"
dest_uuid: "836b43a7ee9146948319b8d37434b630"
from {username='slave'} at 127.0.0.1:50482
I20240503 02:16:27.929368 17764 raft_consensus.cc:482] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:27.929464 17764 raft_consensus.cc:3037] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:16:27.931579 18155 raft_consensus.cc:2454] T 628548a443e64c6fae75070970bba761 P 4325d7d0ed484ad78a6ee54e9a003431 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 2.
I20240503 02:16:27.932286 17764 raft_consensus.cc:504] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:27.932601 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8cc6cfa2b8240c9981a692e0896ef50" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431"
I20240503 02:16:27.932849 18155 raft_consensus.cc:3037] T a8cc6cfa2b8240c9981a692e0896ef50 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.932947 17764 leader_election.cc:288] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 3 election: Requested vote from peers 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703), 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647)
I20240503 02:16:27.934619 18025 raft_consensus.cc:2454] T 628548a443e64c6fae75070970bba761 P 1cd8517d70d94ae494e713df7eb61b12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 2.
I20240503 02:16:27.935889 18155 raft_consensus.cc:2454] T a8cc6cfa2b8240c9981a692e0896ef50 P 4325d7d0ed484ad78a6ee54e9a003431 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 2.
I20240503 02:16:27.936388 18025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac1c650360f4088ba5cc339cee6d45e" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:27.936652 18025 raft_consensus.cc:3037] T 9ac1c650360f4088ba5cc339cee6d45e P 1cd8517d70d94ae494e713df7eb61b12 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:16:27.937462 17894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8cc6cfa2b8240c9981a692e0896ef50" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
I20240503 02:16:27.937692 17894 raft_consensus.cc:3032] T a8cc6cfa2b8240c9981a692e0896ef50 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:16:27.937785 17894 raft_consensus.cc:729] T a8cc6cfa2b8240c9981a692e0896ef50 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:27.938143 17894 consensus_queue.cc:260] T a8cc6cfa2b8240c9981a692e0896ef50 P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:27.938577 17894 raft_consensus.cc:3037] T a8cc6cfa2b8240c9981a692e0896ef50 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.939970 18025 raft_consensus.cc:2454] T 9ac1c650360f4088ba5cc339cee6d45e P 1cd8517d70d94ae494e713df7eb61b12 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 3.
I20240503 02:16:27.940399 17763 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "da0b26fe93614c818d195a45029d5b53"
dest_uuid: "836b43a7ee9146948319b8d37434b630"
from {username='slave'} at 127.0.0.1:50482
I20240503 02:16:27.940603 17763 raft_consensus.cc:465] T da0b26fe93614c818d195a45029d5b53 P 836b43a7ee9146948319b8d37434b630 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 02:16:27.941632 17764 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c5812cc63a1f4aa2b8efdcabb1bf5b29"
dest_uuid: "836b43a7ee9146948319b8d37434b630"
from {username='slave'} at 127.0.0.1:50482
I20240503 02:16:27.941817 17764 raft_consensus.cc:482] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:27.941813 17894 raft_consensus.cc:2454] T a8cc6cfa2b8240c9981a692e0896ef50 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 2.
I20240503 02:16:27.941959 17764 raft_consensus.cc:3037] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:16:27.942889 17894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac1c650360f4088ba5cc339cee6d45e" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
I20240503 02:16:27.943145 17894 raft_consensus.cc:3032] T 9ac1c650360f4088ba5cc339cee6d45e P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:16:27.943293 17894 raft_consensus.cc:729] T 9ac1c650360f4088ba5cc339cee6d45e P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:27.943598 17894 consensus_queue.cc:260] T 9ac1c650360f4088ba5cc339cee6d45e P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:27.943662 17700 leader_election.cc:302] T 628548a443e64c6fae75070970bba761 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cd8517d70d94ae494e713df7eb61b12, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:27.943969 17894 raft_consensus.cc:3037] T 9ac1c650360f4088ba5cc339cee6d45e P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:16:27.945590 17764 raft_consensus.cc:504] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 836b43a7ee9146948319b8d37434b630 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:27.946223 17764 leader_election.cc:288] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 3 election: Requested vote from peers 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647), 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759)
I20240503 02:16:27.946925 17894 raft_consensus.cc:2396] T 9ac1c650360f4088ba5cc339cee6d45e P 0f3838ee7ad04e94aa65ac750dad0833 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 836b43a7ee9146948319b8d37434b630 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:16:27.947155 17762 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7c160a83bd434c56ab99f86d6df68317"
dest_uuid: "836b43a7ee9146948319b8d37434b630"
from {username='slave'} at 127.0.0.1:50482
I20240503 02:16:27.947335 17762 raft_consensus.cc:482] T 7c160a83bd434c56ab99f86d6df68317 P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:27.947475 17762 raft_consensus.cc:3037] T 7c160a83bd434c56ab99f86d6df68317 P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:16:27.948076 17697 leader_election.cc:302] T a8cc6cfa2b8240c9981a692e0896ef50 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:27.950248 17762 raft_consensus.cc:504] T 7c160a83bd434c56ab99f86d6df68317 P 836b43a7ee9146948319b8d37434b630 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:27.950888 17762 leader_election.cc:288] T 7c160a83bd434c56ab99f86d6df68317 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 3 election: Requested vote from peers 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647), 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759)
I20240503 02:16:27.951215 20158 raft_consensus.cc:2781] T 628548a443e64c6fae75070970bba761 P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:16:27.951355 20158 raft_consensus.cc:686] T 628548a443e64c6fae75070970bba761 P 836b43a7ee9146948319b8d37434b630 [term 2 LEADER]: Becoming Leader. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Running, Role: LEADER
I20240503 02:16:27.951673 20158 consensus_queue.cc:237] T 628548a443e64c6fae75070970bba761 P 836b43a7ee9146948319b8d37434b630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:27.952206 17763 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "07ffe3a634c14bd5b0fce34355a5ee4f"
dest_uuid: "836b43a7ee9146948319b8d37434b630"
from {username='slave'} at 127.0.0.1:50482
I20240503 02:16:27.952427 17763 raft_consensus.cc:482] T 07ffe3a634c14bd5b0fce34355a5ee4f P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:27.952564 17763 raft_consensus.cc:3037] T 07ffe3a634c14bd5b0fce34355a5ee4f P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.953192 17700 leader_election.cc:302] T 9ac1c650360f4088ba5cc339cee6d45e P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cd8517d70d94ae494e713df7eb61b12, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:27.953824 19596 raft_consensus.cc:2781] T 9ac1c650360f4088ba5cc339cee6d45e P 836b43a7ee9146948319b8d37434b630 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:16:27.953977 19596 raft_consensus.cc:686] T 9ac1c650360f4088ba5cc339cee6d45e P 836b43a7ee9146948319b8d37434b630 [term 3 LEADER]: Becoming Leader. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Running, Role: LEADER
I20240503 02:16:27.954140 18024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5812cc63a1f4aa2b8efdcabb1bf5b29" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:27.954221 19596 consensus_queue.cc:237] T 9ac1c650360f4088ba5cc339cee6d45e P 836b43a7ee9146948319b8d37434b630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:27.954349 18024 raft_consensus.cc:3037] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 1cd8517d70d94ae494e713df7eb61b12 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:16:27.955271 20160 raft_consensus.cc:2781] T a8cc6cfa2b8240c9981a692e0896ef50 P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:16:27.955394 20160 raft_consensus.cc:686] T a8cc6cfa2b8240c9981a692e0896ef50 P 836b43a7ee9146948319b8d37434b630 [term 2 LEADER]: Becoming Leader. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Running, Role: LEADER
I20240503 02:16:27.955626 20160 consensus_queue.cc:237] T a8cc6cfa2b8240c9981a692e0896ef50 P 836b43a7ee9146948319b8d37434b630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:27.958456 18154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ccff572af764f51938873b46b6622d9" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431"
I20240503 02:16:27.958660 18154 raft_consensus.cc:3037] T 6ccff572af764f51938873b46b6622d9 P 4325d7d0ed484ad78a6ee54e9a003431 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:16:27.961257 17892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5812cc63a1f4aa2b8efdcabb1bf5b29" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
I20240503 02:16:27.961426 17892 raft_consensus.cc:3032] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:16:27.961508 17892 raft_consensus.cc:729] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:27.961735 17892 consensus_queue.cc:260] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:27.962039 17892 raft_consensus.cc:3037] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:16:27.964973 17892 raft_consensus.cc:2396] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 0f3838ee7ad04e94aa65ac750dad0833 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 836b43a7ee9146948319b8d37434b630 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:16:27.965188 17893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c160a83bd434c56ab99f86d6df68317" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
I20240503 02:16:27.965339 17893 raft_consensus.cc:3032] T 7c160a83bd434c56ab99f86d6df68317 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:16:27.965436 17893 raft_consensus.cc:729] T 7c160a83bd434c56ab99f86d6df68317 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:27.965658 17893 consensus_queue.cc:260] T 7c160a83bd434c56ab99f86d6df68317 P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:27.965961 17893 raft_consensus.cc:3037] T 7c160a83bd434c56ab99f86d6df68317 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:16:27.968716 17893 raft_consensus.cc:2396] T 7c160a83bd434c56ab99f86d6df68317 P 0f3838ee7ad04e94aa65ac750dad0833 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 836b43a7ee9146948319b8d37434b630 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:16:27.969341 18023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c160a83bd434c56ab99f86d6df68317" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:27.969571 18023 raft_consensus.cc:3037] T 7c160a83bd434c56ab99f86d6df68317 P 1cd8517d70d94ae494e713df7eb61b12 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:16:27.970402 17616 catalog_manager.cc:5484] T bb6aa5daccb6436fb7910d2d28244b7c P 836b43a7ee9146948319b8d37434b630 reported cstate change: term changed from 1 to 2, leader changed from 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2) to 836b43a7ee9146948319b8d37434b630 (127.22.108.1). New cstate: current_term: 2 leader_uuid: "836b43a7ee9146948319b8d37434b630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } }
I20240503 02:16:27.970861 18154 raft_consensus.cc:2454] T 6ccff572af764f51938873b46b6622d9 P 4325d7d0ed484ad78a6ee54e9a003431 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 3.
I20240503 02:16:27.970922 17616 catalog_manager.cc:5484] T 3fd3dd4b53b5463a8bc8c166ff355ebe P 836b43a7ee9146948319b8d37434b630 reported cstate change: term changed from 2 to 3, leader changed from 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2) to 836b43a7ee9146948319b8d37434b630 (127.22.108.1). New cstate: current_term: 3 leader_uuid: "836b43a7ee9146948319b8d37434b630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } }
I20240503 02:16:27.971355 17697 leader_election.cc:302] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4325d7d0ed484ad78a6ee54e9a003431, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:27.971446 18024 raft_consensus.cc:2454] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 1cd8517d70d94ae494e713df7eb61b12 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 3.
I20240503 02:16:27.971387 17616 catalog_manager.cc:5484] T 353ecf71ac51445087fa3f07e2836db5 P 836b43a7ee9146948319b8d37434b630 reported cstate change: term changed from 2 to 3, leader changed from 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2) to 836b43a7ee9146948319b8d37434b630 (127.22.108.1). New cstate: current_term: 3 leader_uuid: "836b43a7ee9146948319b8d37434b630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } }
I20240503 02:16:27.971675 20158 raft_consensus.cc:2781] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:16:27.971848 20158 raft_consensus.cc:686] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630 [term 3 LEADER]: Becoming Leader. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Running, Role: LEADER
I20240503 02:16:27.971767 17616 catalog_manager.cc:5484] T 954f9a466d8648958df7baa916ea8558 P 836b43a7ee9146948319b8d37434b630 reported cstate change: term changed from 1 to 2, leader changed from 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4) to 836b43a7ee9146948319b8d37434b630 (127.22.108.1). New cstate: current_term: 2 leader_uuid: "836b43a7ee9146948319b8d37434b630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } }
I20240503 02:16:27.972107 20158 consensus_queue.cc:237] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:27.974608 18023 raft_consensus.cc:2454] T 7c160a83bd434c56ab99f86d6df68317 P 1cd8517d70d94ae494e713df7eb61b12 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 3.
I20240503 02:16:27.975929 17894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ccff572af764f51938873b46b6622d9" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
I20240503 02:16:27.976188 17894 raft_consensus.cc:3032] T 6ccff572af764f51938873b46b6622d9 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:16:27.976372 17894 raft_consensus.cc:729] T 6ccff572af764f51938873b46b6622d9 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:27.976702 17894 consensus_queue.cc:260] T 6ccff572af764f51938873b46b6622d9 P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:27.977082 17894 raft_consensus.cc:3037] T 6ccff572af764f51938873b46b6622d9 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:16:27.980180 17894 raft_consensus.cc:2396] T 6ccff572af764f51938873b46b6622d9 P 0f3838ee7ad04e94aa65ac750dad0833 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 836b43a7ee9146948319b8d37434b630 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:16:27.984333 17616 catalog_manager.cc:5484] T 628548a443e64c6fae75070970bba761 P 836b43a7ee9146948319b8d37434b630 reported cstate change: term changed from 1 to 2, leader changed from 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3) to 836b43a7ee9146948319b8d37434b630 (127.22.108.1). New cstate: current_term: 2 leader_uuid: "836b43a7ee9146948319b8d37434b630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } }
I20240503 02:16:27.984838 17700 leader_election.cc:302] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cd8517d70d94ae494e713df7eb61b12, 836b43a7ee9146948319b8d37434b630; no voters: 0f3838ee7ad04e94aa65ac750dad0833
I20240503 02:16:27.984823 17616 catalog_manager.cc:5484] T 6ccff572af764f51938873b46b6622d9 P 836b43a7ee9146948319b8d37434b630 reported cstate change: term changed from 2 to 3, leader changed from 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2) to 836b43a7ee9146948319b8d37434b630 (127.22.108.1). New cstate: current_term: 3 leader_uuid: "836b43a7ee9146948319b8d37434b630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } }
I20240503 02:16:27.985303 17616 catalog_manager.cc:5484] T 9ac1c650360f4088ba5cc339cee6d45e P 836b43a7ee9146948319b8d37434b630 reported cstate change: term changed from 2 to 3, leader changed from 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2) to 836b43a7ee9146948319b8d37434b630 (127.22.108.1). New cstate: current_term: 3 leader_uuid: "836b43a7ee9146948319b8d37434b630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } }
I20240503 02:16:27.985661 17616 catalog_manager.cc:5484] T a8cc6cfa2b8240c9981a692e0896ef50 P 836b43a7ee9146948319b8d37434b630 reported cstate change: term changed from 1 to 2, leader changed from 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2) to 836b43a7ee9146948319b8d37434b630 (127.22.108.1). New cstate: current_term: 2 leader_uuid: "836b43a7ee9146948319b8d37434b630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } }
I20240503 02:16:27.985827 20158 raft_consensus.cc:2781] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 836b43a7ee9146948319b8d37434b630 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:16:27.985966 20158 raft_consensus.cc:686] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 836b43a7ee9146948319b8d37434b630 [term 3 LEADER]: Becoming Leader. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Running, Role: LEADER
I20240503 02:16:27.986231 17700 leader_election.cc:302] T 7c160a83bd434c56ab99f86d6df68317 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cd8517d70d94ae494e713df7eb61b12, 836b43a7ee9146948319b8d37434b630; no voters: 0f3838ee7ad04e94aa65ac750dad0833
I20240503 02:16:27.986325 20158 consensus_queue.cc:237] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 836b43a7ee9146948319b8d37434b630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } }
I20240503 02:16:27.987505 20160 raft_consensus.cc:2781] T 7c160a83bd434c56ab99f86d6df68317 P 836b43a7ee9146948319b8d37434b630 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:16:27.987679 20160 raft_consensus.cc:686] T 7c160a83bd434c56ab99f86d6df68317 P 836b43a7ee9146948319b8d37434b630 [term 3 LEADER]: Becoming Leader. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Running, Role: LEADER
I20240503 02:16:27.987936 20160 consensus_queue.cc:237] T 7c160a83bd434c56ab99f86d6df68317 P 836b43a7ee9146948319b8d37434b630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:27.993530 17763 raft_consensus.cc:504] T 07ffe3a634c14bd5b0fce34355a5ee4f P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:27.994195 17763 leader_election.cc:288] T 07ffe3a634c14bd5b0fce34355a5ee4f P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 2 election: Requested vote from peers 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647), 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703)
I20240503 02:16:27.994864 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ffe3a634c14bd5b0fce34355a5ee4f" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431"
I20240503 02:16:27.995128 18155 raft_consensus.cc:3037] T 07ffe3a634c14bd5b0fce34355a5ee4f P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:27.995220 17764 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6bc5c6531d5144bf80f1b388f23703e9"
dest_uuid: "836b43a7ee9146948319b8d37434b630"
from {username='slave'} at 127.0.0.1:50482
I20240503 02:16:27.995432 17764 raft_consensus.cc:482] T 6bc5c6531d5144bf80f1b388f23703e9 P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:27.995538 17764 raft_consensus.cc:3037] T 6bc5c6531d5144bf80f1b388f23703e9 P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:16:27.998185 18155 raft_consensus.cc:2454] T 07ffe3a634c14bd5b0fce34355a5ee4f P 4325d7d0ed484ad78a6ee54e9a003431 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 2.
I20240503 02:16:27.998221 17764 raft_consensus.cc:504] T 6bc5c6531d5144bf80f1b388f23703e9 P 836b43a7ee9146948319b8d37434b630 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:27.998589 17697 leader_election.cc:302] T 07ffe3a634c14bd5b0fce34355a5ee4f P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4325d7d0ed484ad78a6ee54e9a003431, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:27.998991 20147 raft_consensus.cc:2781] T 07ffe3a634c14bd5b0fce34355a5ee4f P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:16:27.999168 20147 raft_consensus.cc:686] T 07ffe3a634c14bd5b0fce34355a5ee4f P 836b43a7ee9146948319b8d37434b630 [term 2 LEADER]: Becoming Leader. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Running, Role: LEADER
I20240503 02:16:27.999205 18023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bc5c6531d5144bf80f1b388f23703e9" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:27.999426 18023 raft_consensus.cc:3037] T 6bc5c6531d5144bf80f1b388f23703e9 P 1cd8517d70d94ae494e713df7eb61b12 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:16:27.999511 20147 consensus_queue.cc:237] T 07ffe3a634c14bd5b0fce34355a5ee4f P 836b43a7ee9146948319b8d37434b630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:28.000937 17894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ffe3a634c14bd5b0fce34355a5ee4f" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
I20240503 02:16:28.001233 17894 raft_consensus.cc:3032] T 07ffe3a634c14bd5b0fce34355a5ee4f P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:16:28.001322 17894 raft_consensus.cc:729] T 07ffe3a634c14bd5b0fce34355a5ee4f P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:28.001650 17894 consensus_queue.cc:260] T 07ffe3a634c14bd5b0fce34355a5ee4f P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:28.001868 17892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bc5c6531d5144bf80f1b388f23703e9" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
I20240503 02:16:28.002017 17892 raft_consensus.cc:3032] T 6bc5c6531d5144bf80f1b388f23703e9 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:16:28.002081 18023 raft_consensus.cc:2454] T 6bc5c6531d5144bf80f1b388f23703e9 P 1cd8517d70d94ae494e713df7eb61b12 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 3.
I20240503 02:16:28.002107 17892 raft_consensus.cc:729] T 6bc5c6531d5144bf80f1b388f23703e9 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:28.002139 17894 raft_consensus.cc:3037] T 07ffe3a634c14bd5b0fce34355a5ee4f P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:28.002365 17892 consensus_queue.cc:260] T 6bc5c6531d5144bf80f1b388f23703e9 P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:28.002660 17892 raft_consensus.cc:3037] T 6bc5c6531d5144bf80f1b388f23703e9 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:16:28.005884 17892 raft_consensus.cc:2396] T 6bc5c6531d5144bf80f1b388f23703e9 P 0f3838ee7ad04e94aa65ac750dad0833 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 836b43a7ee9146948319b8d37434b630 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:16:28.006491 17700 leader_election.cc:302] T 6bc5c6531d5144bf80f1b388f23703e9 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cd8517d70d94ae494e713df7eb61b12, 836b43a7ee9146948319b8d37434b630; no voters: 0f3838ee7ad04e94aa65ac750dad0833
I20240503 02:16:28.006706 17894 raft_consensus.cc:2454] T 07ffe3a634c14bd5b0fce34355a5ee4f P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 2.
I20240503 02:16:28.006804 20158 raft_consensus.cc:2781] T 6bc5c6531d5144bf80f1b388f23703e9 P 836b43a7ee9146948319b8d37434b630 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:16:28.006959 20158 raft_consensus.cc:686] T 6bc5c6531d5144bf80f1b388f23703e9 P 836b43a7ee9146948319b8d37434b630 [term 3 LEADER]: Becoming Leader. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Running, Role: LEADER
I20240503 02:16:28.007308 20158 consensus_queue.cc:237] T 6bc5c6531d5144bf80f1b388f23703e9 P 836b43a7ee9146948319b8d37434b630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:28.015030 17764 leader_election.cc:288] T 6bc5c6531d5144bf80f1b388f23703e9 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 3 election: Requested vote from peers 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759), 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647)
I20240503 02:16:28.016687 17763 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3d114f397e44d07a729f6eb09f279cc"
dest_uuid: "836b43a7ee9146948319b8d37434b630"
from {username='slave'} at 127.0.0.1:50482
I20240503 02:16:28.016973 17763 raft_consensus.cc:482] T d3d114f397e44d07a729f6eb09f279cc P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:28.017159 17763 raft_consensus.cc:3037] T d3d114f397e44d07a729f6eb09f279cc P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:16:28.021687 17763 raft_consensus.cc:504] T d3d114f397e44d07a729f6eb09f279cc P 836b43a7ee9146948319b8d37434b630 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:28.026015 17763 leader_election.cc:288] T d3d114f397e44d07a729f6eb09f279cc P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 3 election: Requested vote from peers 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647), 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703)
I20240503 02:16:28.027072 17764 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bcd9329d71504749b82738119bf7d3ef"
dest_uuid: "836b43a7ee9146948319b8d37434b630"
from {username='slave'} at 127.0.0.1:50482
I20240503 02:16:28.027323 17764 raft_consensus.cc:482] T bcd9329d71504749b82738119bf7d3ef P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:28.027465 17764 raft_consensus.cc:3037] T bcd9329d71504749b82738119bf7d3ef P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:28.027768 17894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3d114f397e44d07a729f6eb09f279cc" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
I20240503 02:16:28.028002 17894 raft_consensus.cc:3032] T d3d114f397e44d07a729f6eb09f279cc P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:16:28.028095 17894 raft_consensus.cc:729] T d3d114f397e44d07a729f6eb09f279cc P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:28.028378 17894 consensus_queue.cc:260] T d3d114f397e44d07a729f6eb09f279cc P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:28.028712 17894 raft_consensus.cc:3037] T d3d114f397e44d07a729f6eb09f279cc P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:16:28.030354 17764 raft_consensus.cc:504] T bcd9329d71504749b82738119bf7d3ef P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:28.030979 17764 leader_election.cc:288] T bcd9329d71504749b82738119bf7d3ef P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 2 election: Requested vote from peers 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703), 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647)
I20240503 02:16:28.031355 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3d114f397e44d07a729f6eb09f279cc" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431"
I20240503 02:16:28.031575 18155 raft_consensus.cc:3037] T d3d114f397e44d07a729f6eb09f279cc P 4325d7d0ed484ad78a6ee54e9a003431 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:16:28.031896 17762 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "18a22320e5ba41c980e95c1a8d4cf99a"
dest_uuid: "836b43a7ee9146948319b8d37434b630"
from {username='slave'} at 127.0.0.1:50482
I20240503 02:16:28.032070 17762 raft_consensus.cc:482] T 18a22320e5ba41c980e95c1a8d4cf99a P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:28.032161 17762 raft_consensus.cc:3037] T 18a22320e5ba41c980e95c1a8d4cf99a P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:28.034426 18155 raft_consensus.cc:2454] T d3d114f397e44d07a729f6eb09f279cc P 4325d7d0ed484ad78a6ee54e9a003431 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 3.
I20240503 02:16:28.034735 17762 raft_consensus.cc:504] T 18a22320e5ba41c980e95c1a8d4cf99a P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:28.035372 17762 leader_election.cc:288] T 18a22320e5ba41c980e95c1a8d4cf99a P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 2 election: Requested vote from peers 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703), 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759)
I20240503 02:16:28.035907 17616 catalog_manager.cc:5484] T c5812cc63a1f4aa2b8efdcabb1bf5b29 P 836b43a7ee9146948319b8d37434b630 reported cstate change: term changed from 2 to 3, leader changed from 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2) to 836b43a7ee9146948319b8d37434b630 (127.22.108.1). New cstate: current_term: 3 leader_uuid: "836b43a7ee9146948319b8d37434b630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } }
I20240503 02:16:28.036512 17761 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6315543b56a3474da4a13026ad498f84"
dest_uuid: "836b43a7ee9146948319b8d37434b630"
from {username='slave'} at 127.0.0.1:50482
I20240503 02:16:28.036756 17761 raft_consensus.cc:482] T 6315543b56a3474da4a13026ad498f84 P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:28.036500 17616 catalog_manager.cc:5484] T 7c160a83bd434c56ab99f86d6df68317 P 836b43a7ee9146948319b8d37434b630 reported cstate change: term changed from 2 to 3, leader changed from 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2) to 836b43a7ee9146948319b8d37434b630 (127.22.108.1). New cstate: current_term: 3 leader_uuid: "836b43a7ee9146948319b8d37434b630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } }
I20240503 02:16:28.036872 17761 raft_consensus.cc:3037] T 6315543b56a3474da4a13026ad498f84 P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:16:28.037719 17892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd9329d71504749b82738119bf7d3ef" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
I20240503 02:16:28.037935 17892 raft_consensus.cc:3032] T bcd9329d71504749b82738119bf7d3ef P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:16:28.038038 17892 raft_consensus.cc:729] T bcd9329d71504749b82738119bf7d3ef P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:28.038203 17697 leader_election.cc:302] T d3d114f397e44d07a729f6eb09f279cc P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4325d7d0ed484ad78a6ee54e9a003431, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:28.038364 17892 consensus_queue.cc:260] T bcd9329d71504749b82738119bf7d3ef P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:28.038738 17892 raft_consensus.cc:3037] T bcd9329d71504749b82738119bf7d3ef P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:28.040752 18023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a22320e5ba41c980e95c1a8d4cf99a" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:28.040977 18023 raft_consensus.cc:3032] T 18a22320e5ba41c980e95c1a8d4cf99a P 1cd8517d70d94ae494e713df7eb61b12 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:16:28.041102 18023 raft_consensus.cc:729] T 18a22320e5ba41c980e95c1a8d4cf99a P 1cd8517d70d94ae494e713df7eb61b12 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Running, Role: LEADER
I20240503 02:16:28.041282 17892 raft_consensus.cc:2454] T bcd9329d71504749b82738119bf7d3ef P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 2.
I20240503 02:16:28.041376 18023 consensus_queue.cc:260] T 18a22320e5ba41c980e95c1a8d4cf99a P 1cd8517d70d94ae494e713df7eb61b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:28.041786 18023 raft_consensus.cc:3037] T 18a22320e5ba41c980e95c1a8d4cf99a P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:28.041989 17697 leader_election.cc:302] T bcd9329d71504749b82738119bf7d3ef P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:28.042691 17894 raft_consensus.cc:2396] T d3d114f397e44d07a729f6eb09f279cc P 0f3838ee7ad04e94aa65ac750dad0833 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 836b43a7ee9146948319b8d37434b630 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:16:28.043419 20158 raft_consensus.cc:2781] T d3d114f397e44d07a729f6eb09f279cc P 836b43a7ee9146948319b8d37434b630 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:16:28.043650 19596 raft_consensus.cc:2781] T bcd9329d71504749b82738119bf7d3ef P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:16:28.043783 19596 raft_consensus.cc:686] T bcd9329d71504749b82738119bf7d3ef P 836b43a7ee9146948319b8d37434b630 [term 2 LEADER]: Becoming Leader. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Running, Role: LEADER
I20240503 02:16:28.044108 19596 consensus_queue.cc:237] T bcd9329d71504749b82738119bf7d3ef P 836b43a7ee9146948319b8d37434b630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } }
I20240503 02:16:28.044560 18154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a22320e5ba41c980e95c1a8d4cf99a" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431"
I20240503 02:16:28.044770 18154 raft_consensus.cc:3037] T 18a22320e5ba41c980e95c1a8d4cf99a P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:28.045809 20158 raft_consensus.cc:686] T d3d114f397e44d07a729f6eb09f279cc P 836b43a7ee9146948319b8d37434b630 [term 3 LEADER]: Becoming Leader. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Running, Role: LEADER
I20240503 02:16:28.046089 20158 consensus_queue.cc:237] T d3d114f397e44d07a729f6eb09f279cc P 836b43a7ee9146948319b8d37434b630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:28.046347 17621 catalog_manager.cc:5484] T 07ffe3a634c14bd5b0fce34355a5ee4f P 836b43a7ee9146948319b8d37434b630 reported cstate change: term changed from 1 to 2, leader changed from 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2) to 836b43a7ee9146948319b8d37434b630 (127.22.108.1). New cstate: current_term: 2 leader_uuid: "836b43a7ee9146948319b8d37434b630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } }
I20240503 02:16:28.046852 17621 catalog_manager.cc:5484] T 6bc5c6531d5144bf80f1b388f23703e9 P 836b43a7ee9146948319b8d37434b630 reported cstate change: term changed from 2 to 3, leader changed from 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2) to 836b43a7ee9146948319b8d37434b630 (127.22.108.1). New cstate: current_term: 3 leader_uuid: "836b43a7ee9146948319b8d37434b630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } }
I20240503 02:16:28.047622 18154 raft_consensus.cc:2454] T 18a22320e5ba41c980e95c1a8d4cf99a P 4325d7d0ed484ad78a6ee54e9a003431 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 2.
I20240503 02:16:28.049141 18023 raft_consensus.cc:2454] T 18a22320e5ba41c980e95c1a8d4cf99a P 1cd8517d70d94ae494e713df7eb61b12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 2.
I20240503 02:16:28.050880 17697 leader_election.cc:302] T 18a22320e5ba41c980e95c1a8d4cf99a P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4325d7d0ed484ad78a6ee54e9a003431, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:28.051241 20159 raft_consensus.cc:2781] T 18a22320e5ba41c980e95c1a8d4cf99a P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:16:28.051429 20159 raft_consensus.cc:686] T 18a22320e5ba41c980e95c1a8d4cf99a P 836b43a7ee9146948319b8d37434b630 [term 2 LEADER]: Becoming Leader. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Running, Role: LEADER
I20240503 02:16:28.051625 17761 raft_consensus.cc:504] T 6315543b56a3474da4a13026ad498f84 P 836b43a7ee9146948319b8d37434b630 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:28.052685 17761 leader_election.cc:288] T 6315543b56a3474da4a13026ad498f84 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 3 election: Requested vote from peers 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647), 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759)
I20240503 02:16:28.053383 18024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6315543b56a3474da4a13026ad498f84" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:28.053619 18024 raft_consensus.cc:3037] T 6315543b56a3474da4a13026ad498f84 P 1cd8517d70d94ae494e713df7eb61b12 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:16:28.054641 17761 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "13152897990e485c94569e37f1eb92d0"
dest_uuid: "836b43a7ee9146948319b8d37434b630"
from {username='slave'} at 127.0.0.1:50482
I20240503 02:16:28.054829 17761 raft_consensus.cc:465] T 13152897990e485c94569e37f1eb92d0 P 836b43a7ee9146948319b8d37434b630 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 02:16:28.056763 18024 raft_consensus.cc:2454] T 6315543b56a3474da4a13026ad498f84 P 1cd8517d70d94ae494e713df7eb61b12 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 3.
I20240503 02:16:28.057305 17700 leader_election.cc:302] T 6315543b56a3474da4a13026ad498f84 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cd8517d70d94ae494e713df7eb61b12, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:28.057696 20162 raft_consensus.cc:2781] T 6315543b56a3474da4a13026ad498f84 P 836b43a7ee9146948319b8d37434b630 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:16:28.057837 20162 raft_consensus.cc:686] T 6315543b56a3474da4a13026ad498f84 P 836b43a7ee9146948319b8d37434b630 [term 3 LEADER]: Becoming Leader. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Running, Role: LEADER
I20240503 02:16:28.058216 20162 consensus_queue.cc:237] T 6315543b56a3474da4a13026ad498f84 P 836b43a7ee9146948319b8d37434b630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:28.058782 17764 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dacfa7037bae46fb9b3d34ba4b8e54b4"
dest_uuid: "836b43a7ee9146948319b8d37434b630"
from {username='slave'} at 127.0.0.1:50482
I20240503 02:16:28.059370 17764 raft_consensus.cc:482] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:28.059496 17764 raft_consensus.cc:3037] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:16:28.051820 20159 consensus_queue.cc:237] T 18a22320e5ba41c980e95c1a8d4cf99a P 836b43a7ee9146948319b8d37434b630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:28.062680 17764 raft_consensus.cc:504] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 836b43a7ee9146948319b8d37434b630 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:28.062942 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd9329d71504749b82738119bf7d3ef" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431"
I20240503 02:16:28.063267 18155 raft_consensus.cc:3037] T bcd9329d71504749b82738119bf7d3ef P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:28.066797 17894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6315543b56a3474da4a13026ad498f84" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
I20240503 02:16:28.067088 17894 raft_consensus.cc:3032] T 6315543b56a3474da4a13026ad498f84 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:16:28.067214 17894 raft_consensus.cc:729] T 6315543b56a3474da4a13026ad498f84 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:28.067518 17894 consensus_queue.cc:260] T 6315543b56a3474da4a13026ad498f84 P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:28.068486 17894 raft_consensus.cc:3037] T 6315543b56a3474da4a13026ad498f84 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:16:28.069819 18155 raft_consensus.cc:2454] T bcd9329d71504749b82738119bf7d3ef P 4325d7d0ed484ad78a6ee54e9a003431 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 2.
I20240503 02:16:28.070216 17892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacfa7037bae46fb9b3d34ba4b8e54b4" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
I20240503 02:16:28.070407 17892 raft_consensus.cc:3032] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:16:28.070510 17892 raft_consensus.cc:729] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:28.070564 17764 leader_election.cc:288] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 3 election: Requested vote from peers 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647), 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703)
I20240503 02:16:28.070747 17892 consensus_queue.cc:260] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:28.071039 17892 raft_consensus.cc:3037] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:16:28.071560 17894 raft_consensus.cc:2396] T 6315543b56a3474da4a13026ad498f84 P 0f3838ee7ad04e94aa65ac750dad0833 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 836b43a7ee9146948319b8d37434b630 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:16:28.073118 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacfa7037bae46fb9b3d34ba4b8e54b4" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431"
I20240503 02:16:28.073364 18155 raft_consensus.cc:3037] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 4325d7d0ed484ad78a6ee54e9a003431 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:16:28.074400 17892 raft_consensus.cc:2396] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 0f3838ee7ad04e94aa65ac750dad0833 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 836b43a7ee9146948319b8d37434b630 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:16:28.074443 17616 catalog_manager.cc:5484] T d3d114f397e44d07a729f6eb09f279cc P 836b43a7ee9146948319b8d37434b630 reported cstate change: term changed from 2 to 3, leader changed from 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2) to 836b43a7ee9146948319b8d37434b630 (127.22.108.1). New cstate: current_term: 3 leader_uuid: "836b43a7ee9146948319b8d37434b630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } }
I20240503 02:16:28.074995 17764 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c5d3ec79ed6f48b085284b2c06fab3ec"
dest_uuid: "836b43a7ee9146948319b8d37434b630"
from {username='slave'} at 127.0.0.1:50482
I20240503 02:16:28.075176 17764 raft_consensus.cc:482] T c5d3ec79ed6f48b085284b2c06fab3ec P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:28.074995 17616 catalog_manager.cc:5484] T 18a22320e5ba41c980e95c1a8d4cf99a P 836b43a7ee9146948319b8d37434b630 reported cstate change: term changed from 1 to 2, leader changed from 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3) to 836b43a7ee9146948319b8d37434b630 (127.22.108.1). New cstate: current_term: 2 leader_uuid: "836b43a7ee9146948319b8d37434b630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } }
I20240503 02:16:28.075300 17764 raft_consensus.cc:3037] T c5d3ec79ed6f48b085284b2c06fab3ec P 836b43a7ee9146948319b8d37434b630 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:28.075460 17616 catalog_manager.cc:5484] T bcd9329d71504749b82738119bf7d3ef P 836b43a7ee9146948319b8d37434b630 reported cstate change: term changed from 1 to 2, leader changed from 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2) to 836b43a7ee9146948319b8d37434b630 (127.22.108.1). New cstate: current_term: 2 leader_uuid: "836b43a7ee9146948319b8d37434b630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } }
I20240503 02:16:28.076419 18155 raft_consensus.cc:2454] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 4325d7d0ed484ad78a6ee54e9a003431 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 3.
I20240503 02:16:28.076792 17697 leader_election.cc:302] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4325d7d0ed484ad78a6ee54e9a003431, 836b43a7ee9146948319b8d37434b630; no voters: 0f3838ee7ad04e94aa65ac750dad0833
I20240503 02:16:28.078372 17764 raft_consensus.cc:504] T c5d3ec79ed6f48b085284b2c06fab3ec P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:28.078572 19989 heartbeater.cc:503] Master 127.22.108.62:34029 was elected leader, sending a full tablet report...
I20240503 02:16:28.079020 17764 leader_election.cc:288] T c5d3ec79ed6f48b085284b2c06fab3ec P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 2 election: Requested vote from peers 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703), 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3:34759)
I20240503 02:16:28.079622 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d3ec79ed6f48b085284b2c06fab3ec" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431"
I20240503 02:16:28.079730 18024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d3ec79ed6f48b085284b2c06fab3ec" candidate_uuid: "836b43a7ee9146948319b8d37434b630" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
I20240503 02:16:28.079990 18024 raft_consensus.cc:3032] T c5d3ec79ed6f48b085284b2c06fab3ec P 1cd8517d70d94ae494e713df7eb61b12 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:16:28.080083 18024 raft_consensus.cc:729] T c5d3ec79ed6f48b085284b2c06fab3ec P 1cd8517d70d94ae494e713df7eb61b12 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Running, Role: LEADER
I20240503 02:16:28.080013 18155 raft_consensus.cc:3037] T c5d3ec79ed6f48b085284b2c06fab3ec P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:28.080788 20159 raft_consensus.cc:2781] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 836b43a7ee9146948319b8d37434b630 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:16:28.080945 20159 raft_consensus.cc:686] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 836b43a7ee9146948319b8d37434b630 [term 3 LEADER]: Becoming Leader. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Running, Role: LEADER
I20240503 02:16:28.081241 20159 consensus_queue.cc:237] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 836b43a7ee9146948319b8d37434b630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:28.080363 18024 consensus_queue.cc:260] T c5d3ec79ed6f48b085284b2c06fab3ec P 1cd8517d70d94ae494e713df7eb61b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:28.083860 18024 raft_consensus.cc:3037] T c5d3ec79ed6f48b085284b2c06fab3ec P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:28.084118 18155 raft_consensus.cc:2454] T c5d3ec79ed6f48b085284b2c06fab3ec P 4325d7d0ed484ad78a6ee54e9a003431 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 2.
I20240503 02:16:28.084585 17697 leader_election.cc:302] T c5d3ec79ed6f48b085284b2c06fab3ec P 836b43a7ee9146948319b8d37434b630 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4325d7d0ed484ad78a6ee54e9a003431, 836b43a7ee9146948319b8d37434b630; no voters:
I20240503 02:16:28.084863 20158 raft_consensus.cc:2781] T c5d3ec79ed6f48b085284b2c06fab3ec P 836b43a7ee9146948319b8d37434b630 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:16:28.085106 20158 raft_consensus.cc:686] T c5d3ec79ed6f48b085284b2c06fab3ec P 836b43a7ee9146948319b8d37434b630 [term 2 LEADER]: Becoming Leader. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Running, Role: LEADER
I20240503 02:16:28.085394 20158 consensus_queue.cc:237] T c5d3ec79ed6f48b085284b2c06fab3ec P 836b43a7ee9146948319b8d37434b630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:28.086829 18024 raft_consensus.cc:2454] T c5d3ec79ed6f48b085284b2c06fab3ec P 1cd8517d70d94ae494e713df7eb61b12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836b43a7ee9146948319b8d37434b630 in term 2.
I20240503 02:16:28.093711 17616 catalog_manager.cc:5484] T 6315543b56a3474da4a13026ad498f84 P 836b43a7ee9146948319b8d37434b630 reported cstate change: term changed from 2 to 3, leader changed from 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2) to 836b43a7ee9146948319b8d37434b630 (127.22.108.1). New cstate: current_term: 3 leader_uuid: "836b43a7ee9146948319b8d37434b630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } }
I20240503 02:16:28.094327 17616 catalog_manager.cc:5484] T dacfa7037bae46fb9b3d34ba4b8e54b4 P 836b43a7ee9146948319b8d37434b630 reported cstate change: term changed from 2 to 3, leader changed from 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2) to 836b43a7ee9146948319b8d37434b630 (127.22.108.1). New cstate: current_term: 3 leader_uuid: "836b43a7ee9146948319b8d37434b630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } }
I20240503 02:16:28.099735 17616 catalog_manager.cc:5484] T c5d3ec79ed6f48b085284b2c06fab3ec P 836b43a7ee9146948319b8d37434b630 reported cstate change: term changed from 1 to 2, leader changed from 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3) to 836b43a7ee9146948319b8d37434b630 (127.22.108.1). New cstate: current_term: 2 leader_uuid: "836b43a7ee9146948319b8d37434b630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } }
I20240503 02:16:28.109681 18154 raft_consensus.cc:1261] T 0122399369d14e57a55a1415c710e72f P 4325d7d0ed484ad78a6ee54e9a003431 [term 2 FOLLOWER]: Refusing update from remote peer 0f3838ee7ad04e94aa65ac750dad0833: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:16:28.110383 20144 consensus_queue.cc:1035] T 0122399369d14e57a55a1415c710e72f P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:16:28.122054 18155 raft_consensus.cc:1261] T ae8c277b101444af870f4f9fd73d915b P 4325d7d0ed484ad78a6ee54e9a003431 [term 2 FOLLOWER]: Refusing update from remote peer 0f3838ee7ad04e94aa65ac750dad0833: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:16:28.122676 20144 consensus_queue.cc:1035] T ae8c277b101444af870f4f9fd73d915b P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:16:28.126399 18024 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c0300c0a94654f08b08f0ef0cea1d6be"
dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
from {username='slave'} at 127.0.0.1:42096
I20240503 02:16:28.126683 18024 raft_consensus.cc:482] T c0300c0a94654f08b08f0ef0cea1d6be P 1cd8517d70d94ae494e713df7eb61b12 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:28.126824 18024 raft_consensus.cc:3037] T c0300c0a94654f08b08f0ef0cea1d6be P 1cd8517d70d94ae494e713df7eb61b12 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:16:28.128971 18024 raft_consensus.cc:504] T c0300c0a94654f08b08f0ef0cea1d6be P 1cd8517d70d94ae494e713df7eb61b12 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:28.130563 18023 raft_consensus.cc:1261] T 0122399369d14e57a55a1415c710e72f P 1cd8517d70d94ae494e713df7eb61b12 [term 2 FOLLOWER]: Refusing update from remote peer 0f3838ee7ad04e94aa65ac750dad0833: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:16:28.130635 17892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0300c0a94654f08b08f0ef0cea1d6be" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
I20240503 02:16:28.130875 17892 raft_consensus.cc:3032] T c0300c0a94654f08b08f0ef0cea1d6be P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:16:28.131023 17892 raft_consensus.cc:729] T c0300c0a94654f08b08f0ef0cea1d6be P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:28.131366 17892 consensus_queue.cc:260] T c0300c0a94654f08b08f0ef0cea1d6be P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:28.131546 20144 consensus_queue.cc:1035] T 0122399369d14e57a55a1415c710e72f P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:16:28.131763 17892 raft_consensus.cc:3037] T c0300c0a94654f08b08f0ef0cea1d6be P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:16:28.140573 17892 raft_consensus.cc:2396] T c0300c0a94654f08b08f0ef0cea1d6be P 0f3838ee7ad04e94aa65ac750dad0833 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cd8517d70d94ae494e713df7eb61b12 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:16:28.140661 18025 raft_consensus.cc:1261] T ae8c277b101444af870f4f9fd73d915b P 1cd8517d70d94ae494e713df7eb61b12 [term 2 FOLLOWER]: Refusing update from remote peer 0f3838ee7ad04e94aa65ac750dad0833: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:16:28.142313 18153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0300c0a94654f08b08f0ef0cea1d6be" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431"
I20240503 02:16:28.142712 18153 raft_consensus.cc:3037] T c0300c0a94654f08b08f0ef0cea1d6be P 4325d7d0ed484ad78a6ee54e9a003431 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:16:28.150079 18153 raft_consensus.cc:2454] T c0300c0a94654f08b08f0ef0cea1d6be P 4325d7d0ed484ad78a6ee54e9a003431 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cd8517d70d94ae494e713df7eb61b12 in term 3.
I20240503 02:16:28.150537 17958 leader_election.cc:302] T c0300c0a94654f08b08f0ef0cea1d6be P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cd8517d70d94ae494e713df7eb61b12, 4325d7d0ed484ad78a6ee54e9a003431; no voters: 0f3838ee7ad04e94aa65ac750dad0833
I20240503 02:16:28.150878 20133 raft_consensus.cc:2781] T c0300c0a94654f08b08f0ef0cea1d6be P 1cd8517d70d94ae494e713df7eb61b12 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:16:28.151358 20133 raft_consensus.cc:686] T c0300c0a94654f08b08f0ef0cea1d6be P 1cd8517d70d94ae494e713df7eb61b12 [term 3 LEADER]: Becoming Leader. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Running, Role: LEADER
I20240503 02:16:28.151646 20133 consensus_queue.cc:237] T c0300c0a94654f08b08f0ef0cea1d6be P 1cd8517d70d94ae494e713df7eb61b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } }
I20240503 02:16:28.153056 18024 leader_election.cc:288] T c0300c0a94654f08b08f0ef0cea1d6be P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 3 election: Requested vote from peers 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647), 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703)
I20240503 02:16:28.153797 20131 consensus_queue.cc:1035] T ae8c277b101444af870f4f9fd73d915b P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:16:28.154850 18025 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0fda016a9d3b4f238c0ed4bcf47a00b1"
dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
from {username='slave'} at 127.0.0.1:42096
I20240503 02:16:28.155135 18025 raft_consensus.cc:482] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 1cd8517d70d94ae494e713df7eb61b12 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:28.155278 18025 raft_consensus.cc:3037] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 1cd8517d70d94ae494e713df7eb61b12 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:16:28.160233 18025 raft_consensus.cc:504] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 1cd8517d70d94ae494e713df7eb61b12 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:28.161379 18025 leader_election.cc:288] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 4 election: Requested vote from peers 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837), 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647)
I20240503 02:16:28.161984 17764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fda016a9d3b4f238c0ed4bcf47a00b1" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "836b43a7ee9146948319b8d37434b630"
I20240503 02:16:28.162220 17764 raft_consensus.cc:3032] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 836b43a7ee9146948319b8d37434b630 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:16:28.162302 17764 raft_consensus.cc:729] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 836b43a7ee9146948319b8d37434b630 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 836b43a7ee9146948319b8d37434b630, State: Running, Role: LEADER
I20240503 02:16:28.162597 17764 consensus_queue.cc:260] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 836b43a7ee9146948319b8d37434b630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:28.162806 17764 raft_consensus.cc:3037] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 836b43a7ee9146948319b8d37434b630 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:16:28.164733 17764 raft_consensus.cc:2396] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 836b43a7ee9146948319b8d37434b630 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cd8517d70d94ae494e713df7eb61b12 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:16:28.166471 17892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fda016a9d3b4f238c0ed4bcf47a00b1" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
I20240503 02:16:28.166673 17892 raft_consensus.cc:3037] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 0f3838ee7ad04e94aa65ac750dad0833 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:16:28.170506 17892 raft_consensus.cc:2396] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 0f3838ee7ad04e94aa65ac750dad0833 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cd8517d70d94ae494e713df7eb61b12 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 02:16:28.171622 17958 leader_election.cc:302] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cd8517d70d94ae494e713df7eb61b12; no voters: 0f3838ee7ad04e94aa65ac750dad0833, 836b43a7ee9146948319b8d37434b630
I20240503 02:16:28.172580 20133 raft_consensus.cc:2726] T 0fda016a9d3b4f238c0ed4bcf47a00b1 P 1cd8517d70d94ae494e713df7eb61b12 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 02:16:28.172740 18024 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "736e23a32ac944e2825262be0f7d3992"
dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
from {username='slave'} at 127.0.0.1:42096
I20240503 02:16:28.172968 18024 raft_consensus.cc:482] T 736e23a32ac944e2825262be0f7d3992 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:28.173287 18024 raft_consensus.cc:3037] T 736e23a32ac944e2825262be0f7d3992 P 1cd8517d70d94ae494e713df7eb61b12 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:28.175403 17616 catalog_manager.cc:5484] T c0300c0a94654f08b08f0ef0cea1d6be P 1cd8517d70d94ae494e713df7eb61b12 reported cstate change: term changed from 2 to 3, leader changed from 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2) to 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3). New cstate: current_term: 3 leader_uuid: "1cd8517d70d94ae494e713df7eb61b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } }
I20240503 02:16:28.180425 18024 raft_consensus.cc:504] T 736e23a32ac944e2825262be0f7d3992 P 1cd8517d70d94ae494e713df7eb61b12 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:28.182938 17892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736e23a32ac944e2825262be0f7d3992" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0f3838ee7ad04e94aa65ac750dad0833"
I20240503 02:16:28.183245 17892 raft_consensus.cc:3032] T 736e23a32ac944e2825262be0f7d3992 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:16:28.183351 17892 raft_consensus.cc:729] T 736e23a32ac944e2825262be0f7d3992 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0f3838ee7ad04e94aa65ac750dad0833, State: Running, Role: LEADER
I20240503 02:16:28.183704 17892 consensus_queue.cc:260] T 736e23a32ac944e2825262be0f7d3992 P 0f3838ee7ad04e94aa65ac750dad0833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:28.184206 17892 raft_consensus.cc:3037] T 736e23a32ac944e2825262be0f7d3992 P 0f3838ee7ad04e94aa65ac750dad0833 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:28.186471 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736e23a32ac944e2825262be0f7d3992" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4325d7d0ed484ad78a6ee54e9a003431"
I20240503 02:16:28.186848 18155 raft_consensus.cc:3037] T 736e23a32ac944e2825262be0f7d3992 P 4325d7d0ed484ad78a6ee54e9a003431 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:28.188007 17892 raft_consensus.cc:2454] T 736e23a32ac944e2825262be0f7d3992 P 0f3838ee7ad04e94aa65ac750dad0833 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cd8517d70d94ae494e713df7eb61b12 in term 2.
I20240503 02:16:28.188611 17958 leader_election.cc:302] T 736e23a32ac944e2825262be0f7d3992 P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3838ee7ad04e94aa65ac750dad0833, 1cd8517d70d94ae494e713df7eb61b12; no voters:
I20240503 02:16:28.189074 20133 raft_consensus.cc:2781] T 736e23a32ac944e2825262be0f7d3992 P 1cd8517d70d94ae494e713df7eb61b12 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:16:28.189249 20133 raft_consensus.cc:686] T 736e23a32ac944e2825262be0f7d3992 P 1cd8517d70d94ae494e713df7eb61b12 [term 2 LEADER]: Becoming Leader. State: Replica: 1cd8517d70d94ae494e713df7eb61b12, State: Running, Role: LEADER
I20240503 02:16:28.189559 20133 consensus_queue.cc:237] T 736e23a32ac944e2825262be0f7d3992 P 1cd8517d70d94ae494e713df7eb61b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:28.193329 18155 raft_consensus.cc:2454] T 736e23a32ac944e2825262be0f7d3992 P 4325d7d0ed484ad78a6ee54e9a003431 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cd8517d70d94ae494e713df7eb61b12 in term 2.
I20240503 02:16:28.193661 18152 raft_consensus.cc:1261] T 0f8c3cd667594af4969548ba820e0164 P 4325d7d0ed484ad78a6ee54e9a003431 [term 2 FOLLOWER]: Refusing update from remote peer 0f3838ee7ad04e94aa65ac750dad0833: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:16:28.194255 20131 consensus_queue.cc:1035] T 0f8c3cd667594af4969548ba820e0164 P 0f3838ee7ad04e94aa65ac750dad0833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:16:28.196732 17616 catalog_manager.cc:5484] T 736e23a32ac944e2825262be0f7d3992 P 1cd8517d70d94ae494e713df7eb61b12 reported cstate change: term changed from 1 to 2, leader changed from 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2) to 1cd8517d70d94ae494e713df7eb61b12 (127.22.108.3). New cstate: current_term: 2 leader_uuid: "1cd8517d70d94ae494e713df7eb61b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4325d7d0ed484ad78a6ee54e9a003431" member_type: VOTER last_known_addr { host: "127.22.108.4" port: 39703 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } } }
I20240503 02:16:28.201632 18024 leader_election.cc:288] T 736e23a32ac944e2825262be0f7d3992 P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 2 election: Requested vote from peers 4325d7d0ed484ad78a6ee54e9a003431 (127.22.108.4:39703), 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647)
I20240503 02:16:28.203065 18024 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bb6aa5daccb6436fb7910d2d28244b7c"
dest_uuid: "1cd8517d70d94ae494e713df7eb61b12"
from {username='slave'} at 127.0.0.1:42096
I20240503 02:16:28.203318 18024 raft_consensus.cc:482] T bb6aa5daccb6436fb7910d2d28244b7c P 1cd8517d70d94ae494e713df7eb61b12 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:28.203436 18024 raft_consensus.cc:3037] T bb6aa5daccb6436fb7910d2d28244b7c P 1cd8517d70d94ae494e713df7eb61b12 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:16:28.205369 18024 raft_consensus.cc:504] T bb6aa5daccb6436fb7910d2d28244b7c P 1cd8517d70d94ae494e713df7eb61b12 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836b43a7ee9146948319b8d37434b630" member_type: VOTER last_known_addr { host: "127.22.108.1" port: 45837 } } peers { permanent_uuid: "0f3838ee7ad04e94aa65ac750dad0833" member_type: VOTER last_known_addr { host: "127.22.108.2" port: 41647 } } peers { permanent_uuid: "1cd8517d70d94ae494e713df7eb61b12" member_type: VOTER last_known_addr { host: "127.22.108.3" port: 34759 } }
I20240503 02:16:28.206049 18024 leader_election.cc:288] T bb6aa5daccb6436fb7910d2d28244b7c P 1cd8517d70d94ae494e713df7eb61b12 [CANDIDATE]: Term 3 election: Requested vote from peers 836b43a7ee9146948319b8d37434b630 (127.22.108.1:45837), 0f3838ee7ad04e94aa65ac750dad0833 (127.22.108.2:41647)
I20240503 02:16:28.206760 17764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb6aa5daccb6436fb7910d2d28244b7c" candidate_uuid: "1cd8517d70d94ae494e713df7eb61b12" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "836b43a7ee9146948319b8d37434b630"